2024-09-13T11:46:26.273 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-13T11:46:26.286 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T11:46:26.399 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903460 branch: wip-aclamk-cbt-combined description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd} email: akupczyk@redhat.com first_in_suite: false job_id: '7903460' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-aclamk-cbt-combined ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 workunit: branch: wip-aclamk-cbt-combined sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 owner: scheduled_akupczyk@teuthology priority: 160 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 4159 sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 sleep_before_teardown: 0 subset: 682/13280 suite: rados suite_branch: wip-aclamk-cbt-combined suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 targets: smithi032.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBFv6veMBSR6uISJDQaUvkMbH/iVIWtyJRdOxBwJ2tMbAkGtu+R3WdyDQBCc5+MhwVrCgUS+2c7GZ9M1+2eA9Ig= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-13_09:43:51 tube: smithi user: akupczyk verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-13T11:46:26.399 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa; will attempt to use it 2024-09-13T11:46:26.400 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks 2024-09-13T11:46:26.400 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-13T11:46:26.402 INFO:teuthology.task.internal:Checking packages... 2024-09-13T11:46:26.426 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd58f73069a8157a9ace12a0247ee3e8497707ea9' 2024-09-13T11:46:26.426 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T11:46:26.426 INFO:teuthology.packaging:ref: None 2024-09-13T11:46:26.426 INFO:teuthology.packaging:tag: None 2024-09-13T11:46:26.426 INFO:teuthology.packaging:branch: wip-aclamk-cbt-combined 2024-09-13T11:46:26.426 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:46:26.426 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-aclamk-cbt-combined 2024-09-13T11:46:26.589 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4697.gd58f7306 2024-09-13T11:46:26.591 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-13T11:46:26.596 INFO:teuthology.task.internal:no buildpackages task found 2024-09-13T11:46:26.596 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-13T11:46:26.618 INFO:teuthology.task.internal:Saving configuration 2024-09-13T11:46:26.628 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-13T11:46:26.634 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-13T11:46:26.667 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903460', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 11:41:52.752784', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBFv6veMBSR6uISJDQaUvkMbH/iVIWtyJRdOxBwJ2tMbAkGtu+R3WdyDQBCc5+MhwVrCgUS+2c7GZ9M1+2eA9Ig='} 2024-09-13T11:46:26.667 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-13T11:46:26.673 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-13T11:46:26.673 INFO:teuthology.run_tasks:Running task console_log... 2024-09-13T11:46:26.718 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f418b33bbe0>, signals=[15]) 2024-09-13T11:46:26.718 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-13T11:46:26.723 INFO:teuthology.task.internal:Opening connections... 2024-09-13T11:46:26.724 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi032.front.sepia.ceph.com 2024-09-13T11:46:26.725 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:46:26.835 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-13T11:46:26.841 DEBUG:teuthology.orchestra.run.smithi032:> uname -m 2024-09-13T11:46:26.923 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64 2024-09-13T11:46:26.923 DEBUG:teuthology.orchestra.run.smithi032:> cat /etc/os-release 2024-09-13T11:46:26.981 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="CentOS Stream" 2024-09-13T11:46:26.981 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="9" 2024-09-13T11:46:26.981 INFO:teuthology.orchestra.run.smithi032.stdout:ID="centos" 2024-09-13T11:46:26.981 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE="rhel fedora" 2024-09-13T11:46:26.981 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="9" 2024-09-13T11:46:26.981 INFO:teuthology.orchestra.run.smithi032.stdout:PLATFORM_ID="platform:el9" 2024-09-13T11:46:26.981 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T11:46:26.981 INFO:teuthology.orchestra.run.smithi032.stdout:ANSI_COLOR="0;31" 2024-09-13T11:46:26.981 INFO:teuthology.orchestra.run.smithi032.stdout:LOGO="fedora-logo-icon" 2024-09-13T11:46:26.982 INFO:teuthology.orchestra.run.smithi032.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T11:46:26.982 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://centos.org/" 2024-09-13T11:46:26.982 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T11:46:26.982 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T11:46:26.982 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T11:46:26.982 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2024-09-13T11:46:27.010 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-13T11:46:27.018 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-13T11:46:27.023 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-13T11:46:27.023 DEBUG:teuthology.orchestra.run.smithi032:> test '!' -e /home/ubuntu/cephtest 2024-09-13T11:46:27.040 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-13T11:46:27.046 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-13T11:46:27.046 DEBUG:teuthology.orchestra.run.smithi032:> test -z $(ls -A /var/lib/ceph) 2024-09-13T11:46:27.102 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-13T11:46:27.142 INFO:teuthology.run_tasks:Running task kernel... 2024-09-13T11:46:27.157 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-13T11:46:27.157 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-13T11:46:27.158 DEBUG:teuthology.orchestra.run.smithi032:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T11:46:27.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:46:27.176 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-09-13T11:46:27.235 INFO:teuthology.orchestra.run.smithi032.stdout:5.14.0-503.el9.x86_64 2024-09-13T11:46:27.235 INFO:teuthology.task.kernel:Running kernel on smithi032: 5.14.0-503.el9.x86_64 2024-09-13T11:46:27.235 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum install -y kernel 2024-09-13T11:46:29.540 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.2 MB 00:00 2024-09-13T11:46:32.899 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - AppStream 9.3 MB/s | 20 MB 00:02 2024-09-13T11:46:37.317 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-13T11:46:38.901 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 19 kB 00:00 2024-09-13T11:46:39.996 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-13T11:46:45.711 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-09-13T11:46:47.743 INFO:teuthology.orchestra.run.smithi032.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T11:46:47.743 INFO:teuthology.orchestra.run.smithi032.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T11:46:47.840 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-13T11:46:47.846 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:46:47.847 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-09-13T11:46:47.847 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:46:47.847 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-09-13T11:46:47.847 INFO:teuthology.orchestra.run.smithi032.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T11:46:47.847 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-09-13T11:46:47.847 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T11:46:47.847 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T11:46:47.847 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T11:46:47.847 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:46:47.847 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-09-13T11:46:47.847 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:46:47.847 INFO:teuthology.orchestra.run.smithi032.stdout:Install 4 Packages 2024-09-13T11:46:47.848 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:46:47.848 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 79 M 2024-09-13T11:46:47.848 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 126 M 2024-09-13T11:46:47.848 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-09-13T11:46:48.071 INFO:teuthology.orchestra.run.smithi032.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 276 kB/s | 23 kB 00:00 2024-09-13T11:46:50.679 INFO:teuthology.orchestra.run.smithi032.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 5.8 MB/s | 16 MB 00:02 2024-09-13T11:46:52.254 INFO:teuthology.orchestra.run.smithi032.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 6.8 MB/s | 29 MB 00:04 2024-09-13T11:46:52.733 INFO:teuthology.orchestra.run.smithi032.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 7.3 MB/s | 35 MB 00:04 2024-09-13T11:46:52.733 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:46:52.733 INFO:teuthology.orchestra.run.smithi032.stdout:Total 16 MB/s | 79 MB 00:04 2024-09-13T11:46:53.443 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-09-13T11:46:53.832 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-09-13T11:46:53.833 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-09-13T11:46:54.817 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-09-13T11:46:54.817 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-09-13T11:46:57.403 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-09-13T11:46:58.099 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T11:46:58.181 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:47:00.973 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:47:01.102 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:47:06.227 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:47:06.297 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:47:11.442 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:47:50.639 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:47:50.681 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:47:51.050 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:47:51.050 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T11:47:51.050 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:47:51.051 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:47:51.890 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:47:51.891 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:47:51.891 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-09-13T11:47:51.891 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T11:47:51.891 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T11:47:51.891 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T11:47:51.891 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T11:47:51.891 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:47:51.891 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-13T11:47:52.267 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T11:47:52.353 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T11:47:52.353 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T11:47:52.353 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T11:47:52.353 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T11:47:52.354 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T11:47:52.354 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-13T11:47:52.354 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi032.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T11:47:52.354 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T11:47:52.412 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T11:47:52.412 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T11:47:52.412 DEBUG:teuthology.orchestra.run.smithi032:> sudo rpm -qi grub2-tools 2024-09-13T11:47:52.443 INFO:teuthology.orchestra.run.smithi032.stdout:Name : grub2-tools 2024-09-13T11:47:52.443 INFO:teuthology.orchestra.run.smithi032.stdout:Epoch : 1 2024-09-13T11:47:52.443 INFO:teuthology.orchestra.run.smithi032.stdout:Version : 2.06 2024-09-13T11:47:52.443 INFO:teuthology.orchestra.run.smithi032.stdout:Release : 82.el9 2024-09-13T11:47:52.443 INFO:teuthology.orchestra.run.smithi032.stdout:Architecture: x86_64 2024-09-13T11:47:52.443 INFO:teuthology.orchestra.run.smithi032.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T11:47:52.444 INFO:teuthology.orchestra.run.smithi032.stdout:Group : Unspecified 2024-09-13T11:47:52.444 INFO:teuthology.orchestra.run.smithi032.stdout:Size : 8274670 2024-09-13T11:47:52.444 INFO:teuthology.orchestra.run.smithi032.stdout:License : GPLv3+ 2024-09-13T11:47:52.444 INFO:teuthology.orchestra.run.smithi032.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T11:47:52.444 INFO:teuthology.orchestra.run.smithi032.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T11:47:52.444 INFO:teuthology.orchestra.run.smithi032.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T11:47:52.444 INFO:teuthology.orchestra.run.smithi032.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T11:47:52.444 INFO:teuthology.orchestra.run.smithi032.stdout:Packager : builder@centos.org 2024-09-13T11:47:52.444 INFO:teuthology.orchestra.run.smithi032.stdout:Vendor : CentOS 2024-09-13T11:47:52.444 INFO:teuthology.orchestra.run.smithi032.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T11:47:52.444 INFO:teuthology.orchestra.run.smithi032.stdout:Summary : Support tools for GRUB. 2024-09-13T11:47:52.444 INFO:teuthology.orchestra.run.smithi032.stdout:Description : 2024-09-13T11:47:52.444 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:47:52.445 INFO:teuthology.orchestra.run.smithi032.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T11:47:52.445 INFO:teuthology.orchestra.run.smithi032.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T11:47:52.445 INFO:teuthology.orchestra.run.smithi032.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T11:47:52.445 INFO:teuthology.orchestra.run.smithi032.stdout:hardware devices. 2024-09-13T11:47:52.445 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:47:52.445 INFO:teuthology.orchestra.run.smithi032.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T11:47:52.446 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T11:47:52.446 INFO:teuthology.task.kernel:Updating grub on smithi032 to boot 5.14.0-505.el9.x86_64 2024-09-13T11:47:52.446 DEBUG:teuthology.orchestra.run.smithi032:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T11:47:53.128 INFO:teuthology.orchestra.run.smithi032.stderr:Generating grub configuration file ... 2024-09-13T11:47:54.447 INFO:teuthology.orchestra.run.smithi032.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T11:47:54.475 INFO:teuthology.orchestra.run.smithi032.stderr:done 2024-09-13T11:47:54.482 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2024-09-13T11:47:54.497 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.125MMMTsG5 2024-09-13T11:47:54.498 DEBUG:teuthology.orchestra.run.smithi032:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.125MMMTsG5 2024-09-13T11:47:54.565 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.125MMMTsG5 2024-09-13T11:47:54.704 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.125MMMTsG5 is 6KB 2024-09-13T11:47:54.753 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.125MMMTsG5 2024-09-13T11:47:54.768 DEBUG:teuthology.orchestra.run.smithi032:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T11:47:54.834 INFO:teuthology.orchestra.run.smithi032.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T11:47:54.834 INFO:teuthology.orchestra.run.smithi032.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T11:47:54.834 INFO:teuthology.orchestra.run.smithi032.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T11:47:54.834 INFO:teuthology.orchestra.run.smithi032.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T11:47:54.835 DEBUG:teuthology.orchestra.run.smithi032:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T11:47:54.971 DEBUG:teuthology.orchestra.run.smithi032:> sudo shutdown -r now 2024-09-13T11:48:25.002 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-13T11:48:25.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-09-13T11:48:25.004 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:48:43.453 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.32 2024-09-13T11:48:52.461 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-09-13T11:48:52.462 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:49:10.845 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.32 2024-09-13T11:49:22.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-09-13T11:49:22.859 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:49:23.297 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-09-13T11:49:23.695 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-09-13T11:49:23.695 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-13T11:49:23.695 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-09-13T11:49:23.752 INFO:teuthology.orchestra.run.smithi032.stdout:5.14.0-505.el9.x86_64 2024-09-13T11:49:23.752 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T11:49:23.753 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T11:49:23.753 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T11:49:24.753 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-13T11:49:24.753 DEBUG:teuthology.orchestra.run.smithi032:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T11:49:24.822 INFO:teuthology.orchestra.run.smithi032.stdout:ttyS1 2024-09-13T11:49:24.843 DEBUG:teuthology.parallel:result is None 2024-09-13T11:49:24.843 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-13T11:49:24.851 INFO:teuthology.task.internal:Creating test directory... 2024-09-13T11:49:24.851 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T11:49:24.866 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-13T11:49:24.876 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-13T11:49:24.882 INFO:teuthology.task.internal:Creating archive directory... 2024-09-13T11:49:24.882 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T11:49:24.938 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-13T11:49:24.944 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-13T11:49:24.945 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T11:49:25.011 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:49:25.024 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:49:25.026 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-13T11:49:25.032 INFO:teuthology.task.internal:Configuring sudo... 2024-09-13T11:49:25.033 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T11:49:25.092 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-13T11:49:25.101 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-13T11:49:25.102 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T11:49:25.149 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T11:49:25.242 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T11:49:25.282 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T11:49:25.373 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T11:49:25.408 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-09-13T11:49:25.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T11:49:25.479 DEBUG:teuthology.orchestra.run.smithi032:> sudo service rsyslog restart 2024-09-13T11:49:25.580 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T11:49:26.077 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-13T11:49:26.086 INFO:teuthology.task.internal:Starting timer... 2024-09-13T11:49:26.086 INFO:teuthology.run_tasks:Running task pcp... 2024-09-13T11:49:26.099 INFO:teuthology.run_tasks:Running task selinux... 2024-09-13T11:49:26.112 DEBUG:teuthology.orchestra.run.smithi032:> sudo service auditd rotate 2024-09-13T11:49:26.155 INFO:teuthology.orchestra.run.smithi032.stdout:Rotating logs: 2024-09-13T11:49:26.157 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-13T11:49:26.158 DEBUG:teuthology.orchestra.run.smithi032:> /usr/sbin/getenforce 2024-09-13T11:49:26.232 INFO:teuthology.orchestra.run.smithi032.stdout:Permissive 2024-09-13T11:49:26.233 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi032.front.sepia.ceph.com': 'permissive'} 2024-09-13T11:49:26.233 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T11:49:26.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:49:26.299 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-13T11:49:26.300 DEBUG:teuthology.orchestra.run.smithi032:> sudo /usr/sbin/setenforce permissive 2024-09-13T11:49:26.377 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-13T11:49:26.389 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-13T11:49:26.394 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-13T11:49:26.538 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-13T11:49:26.550 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-13T11:49:26.551 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi032.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-13T11:53:18.033 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi032.front.sepia.ceph.com')] 2024-09-13T11:53:18.034 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-09-13T11:53:18.036 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:53:18.123 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-09-13T11:53:18.205 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-09-13T11:53:18.205 INFO:teuthology.run_tasks:Running task clock... 2024-09-13T11:53:18.219 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-13T11:53:18.220 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T11:53:18.220 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T11:53:18.275 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T11:53:18.291 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T11:53:18.318 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: ntpd: command not found 2024-09-13T11:53:18.329 INFO:teuthology.orchestra.run.smithi032.stdout:506 Cannot talk to daemon 2024-09-13T11:53:18.344 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T11:53:18.359 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T11:53:18.416 INFO:teuthology.orchestra.run.smithi032.stderr:bash: line 1: ntpq: command not found 2024-09-13T11:53:18.420 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T11:53:18.420 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2024-09-13T11:53:18.421 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:53:18.421 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:53:18.421 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:53:18.421 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:53:18.422 INFO:teuthology.run_tasks:Running task install... 2024-09-13T11:53:18.430 DEBUG:teuthology.task.install:project ceph 2024-09-13T11:53:18.430 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}} 2024-09-13T11:53:18.430 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'} 2024-09-13T11:53:18.430 INFO:teuthology.task.install:Using flavor: default 2024-09-13T11:53:18.436 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-13T11:53:18.436 INFO:teuthology.task.install:extra packages: [] 2024-09-13T11:53:18.437 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'tag': None, 'wait_for_package': False} 2024-09-13T11:53:18.437 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:53:18.590 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/ 2024-09-13T11:53:18.591 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4697.gd58f7306 2024-09-13T11:53:18.715 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T11:53:18.715 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-09-13T11:53:18.715 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T11:53:18.744 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-13T11:53:18.744 DEBUG:teuthology.orchestra.run.smithi032:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d58f73069a8157a9ace12a0247ee3e8497707ea9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T11:53:18.812 DEBUG:teuthology.orchestra.run.smithi032:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T11:53:18.889 DEBUG:teuthology.orchestra.run.smithi032:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T11:53:18.955 INFO:teuthology.orchestra.run.smithi032.stdout:check_obsoletes = 1 2024-09-13T11:53:18.957 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum clean all 2024-09-13T11:53:19.210 INFO:teuthology.orchestra.run.smithi032.stdout:45 files removed 2024-09-13T11:53:19.237 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-radosgw 2024-09-13T11:53:19.801 INFO:teuthology.orchestra.run.smithi032.stdout:ceph packages for x86_64 318 kB/s | 84 kB 00:00 2024-09-13T11:53:20.094 INFO:teuthology.orchestra.run.smithi032.stdout:ceph noarch packages 98 kB/s | 24 kB 00:00 2024-09-13T11:53:20.377 INFO:teuthology.orchestra.run.smithi032.stdout:ceph source packages 7.7 kB/s | 1.9 kB 00:00 2024-09-13T11:53:20.819 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - BaseOS 20 MB/s | 8.2 MB 00:00 2024-09-13T11:53:24.096 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - AppStream 9.6 MB/s | 20 MB 00:02 2024-09-13T11:53:29.440 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - CRB 4.9 MB/s | 6.5 MB 00:01 2024-09-13T11:53:30.899 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - Extras packages 101 kB/s | 19 kB 00:00 2024-09-13T11:53:31.803 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-13T11:53:37.393 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-09-13T11:53:39.864 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-13T11:53:39.867 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:53:39.867 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-09-13T11:53:39.867 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:53:39.868 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-09-13T11:53:39.868 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 11 M 2024-09-13T11:53:39.868 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrading: 2024-09-13T11:53:39.868 INFO:teuthology.orchestra.run.smithi032.stdout: librados2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.5 M 2024-09-13T11:53:39.868 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T11:53:39.868 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-09-13T11:53:39.868 INFO:teuthology.orchestra.run.smithi032.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T11:53:39.868 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T11:53:39.868 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 22 M 2024-09-13T11:53:39.868 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 25 k 2024-09-13T11:53:39.868 INFO:teuthology.orchestra.run.smithi032.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T11:53:39.868 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T11:53:39.869 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T11:53:39.869 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 755 k 2024-09-13T11:53:39.869 INFO:teuthology.orchestra.run.smithi032.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T11:53:39.869 INFO:teuthology.orchestra.run.smithi032.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T11:53:39.869 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T11:53:39.869 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 508 k 2024-09-13T11:53:39.869 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T11:53:39.869 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T11:53:39.869 INFO:teuthology.orchestra.run.smithi032.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T11:53:39.869 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T11:53:39.869 INFO:teuthology.orchestra.run.smithi032.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T11:53:39.869 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 45 k 2024-09-13T11:53:39.870 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 143 k 2024-09-13T11:53:39.870 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T11:53:39.870 INFO:teuthology.orchestra.run.smithi032.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T11:53:39.870 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 321 k 2024-09-13T11:53:39.870 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 305 k 2024-09-13T11:53:39.870 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 99 k 2024-09-13T11:53:39.870 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-13T11:53:39.870 INFO:teuthology.orchestra.run.smithi032.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-13T11:53:39.870 INFO:teuthology.orchestra.run.smithi032.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T11:53:39.870 INFO:teuthology.orchestra.run.smithi032.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T11:53:39.870 INFO:teuthology.orchestra.run.smithi032.stdout:Installing weak dependencies: 2024-09-13T11:53:39.870 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-13T11:53:39.871 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:53:39.871 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-09-13T11:53:39.871 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:53:39.871 INFO:teuthology.orchestra.run.smithi032.stdout:Install 30 Packages 2024-09-13T11:53:39.871 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrade 2 Packages 2024-09-13T11:53:39.871 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:53:39.871 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 62 M 2024-09-13T11:53:39.871 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-09-13T11:53:40.777 INFO:teuthology.orchestra.run.smithi032.stdout:(1/32): ceph-base-19.3.0-4697.gd58f7306.el9.x86 13 MB/s | 5.5 MB 00:00 2024-09-13T11:53:40.836 INFO:teuthology.orchestra.run.smithi032.stdout:(2/32): ceph-selinux-19.3.0-4697.gd58f7306.el9. 432 kB/s | 25 kB 00:00 2024-09-13T11:53:40.928 INFO:teuthology.orchestra.run.smithi032.stdout:(3/32): libcephfs2-19.3.0-4697.gd58f7306.el9.x8 8.1 MB/s | 755 kB 00:00 2024-09-13T11:53:41.011 INFO:teuthology.orchestra.run.smithi032.stdout:(4/32): libradosstriper1-19.3.0-4697.gd58f7306. 6.0 MB/s | 508 kB 00:00 2024-09-13T11:53:41.128 INFO:teuthology.orchestra.run.smithi032.stdout:(5/32): ceph-radosgw-19.3.0-4697.gd58f7306.el9. 14 MB/s | 11 MB 00:00 2024-09-13T11:53:41.170 INFO:teuthology.orchestra.run.smithi032.stdout:(6/32): python3-ceph-argparse-19.3.0-4697.gd58f 1.1 MB/s | 45 kB 00:00 2024-09-13T11:53:41.262 INFO:teuthology.orchestra.run.smithi032.stdout:(7/32): librgw2-19.3.0-4697.gd58f7306.el9.x86_6 22 MB/s | 5.5 MB 00:00 2024-09-13T11:53:41.287 INFO:teuthology.orchestra.run.smithi032.stdout:(8/32): python3-ceph-common-19.3.0-4697.gd58f73 1.2 MB/s | 143 kB 00:00 2024-09-13T11:53:41.312 INFO:teuthology.orchestra.run.smithi032.stdout:(9/32): python3-cephfs-19.3.0-4697.gd58f7306.el 3.1 MB/s | 161 kB 00:00 2024-09-13T11:53:41.346 INFO:teuthology.orchestra.run.smithi032.stdout:(10/32): python3-rados-19.3.0-4697.gd58f7306.el 5.3 MB/s | 321 kB 00:00 2024-09-13T11:53:41.380 INFO:teuthology.orchestra.run.smithi032.stdout:(11/32): python3-rbd-19.3.0-4697.gd58f7306.el9. 4.5 MB/s | 305 kB 00:00 2024-09-13T11:53:41.413 INFO:teuthology.orchestra.run.smithi032.stdout:(12/32): python3-rgw-19.3.0-4697.gd58f7306.el9. 1.5 MB/s | 99 kB 00:00 2024-09-13T11:53:41.680 INFO:teuthology.orchestra.run.smithi032.stdout:(13/32): ceph-common-19.3.0-4697.gd58f7306.el9. 16 MB/s | 22 MB 00:01 2024-09-13T11:53:41.747 INFO:teuthology.orchestra.run.smithi032.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 291 kB/s | 107 kB 00:00 2024-09-13T11:53:41.839 INFO:teuthology.orchestra.run.smithi032.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 376 kB/s | 160 kB 00:00 2024-09-13T11:53:41.881 INFO:teuthology.orchestra.run.smithi032.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 226 kB/s | 45 kB 00:00 2024-09-13T11:53:42.006 INFO:teuthology.orchestra.run.smithi032.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 338 kB/s | 42 kB 00:00 2024-09-13T11:53:42.090 INFO:teuthology.orchestra.run.smithi032.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.1 MB/s | 292 kB 00:00 2024-09-13T11:53:42.165 INFO:teuthology.orchestra.run.smithi032.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 479 kB/s | 36 kB 00:00 2024-09-13T11:53:42.199 INFO:teuthology.orchestra.run.smithi032.stdout:(20/32): librdkafka-1.6.1-102.el9.x86_64.rpm 1.4 MB/s | 662 kB 00:00 2024-09-13T11:53:42.232 INFO:teuthology.orchestra.run.smithi032.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 709 kB/s | 47 kB 00:00 2024-09-13T11:53:42.258 INFO:teuthology.orchestra.run.smithi032.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 880 kB/s | 221 kB 00:00 2024-09-13T11:53:42.308 INFO:teuthology.orchestra.run.smithi032.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-13T11:53:42.325 INFO:teuthology.orchestra.run.smithi032.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 3.3 MB/s | 49 kB 00:00 2024-09-13T11:53:42.350 INFO:teuthology.orchestra.run.smithi032.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 311 kB/s | 26 kB 00:00 2024-09-13T11:53:42.375 INFO:teuthology.orchestra.run.smithi032.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-13T11:53:42.401 INFO:teuthology.orchestra.run.smithi032.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 7.5 MB/s | 191 kB 00:00 2024-09-13T11:53:42.476 INFO:teuthology.orchestra.run.smithi032.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-09-13T11:53:42.510 INFO:teuthology.orchestra.run.smithi032.stdout:(29/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 5.1 MB/s | 839 kB 00:00 2024-09-13T11:53:42.593 INFO:teuthology.orchestra.run.smithi032.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-13T11:53:42.727 INFO:teuthology.orchestra.run.smithi032.stdout:(31/32): librados2-19.3.0-4697.gd58f7306.el9.x8 14 MB/s | 3.5 MB 00:00 2024-09-13T11:53:42.785 INFO:teuthology.orchestra.run.smithi032.stdout:(32/32): librbd1-19.3.0-4697.gd58f7306.el9.x86_ 12 MB/s | 3.2 MB 00:00 2024-09-13T11:53:42.791 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:53:42.791 INFO:teuthology.orchestra.run.smithi032.stdout:Total 21 MB/s | 62 MB 00:02 2024-09-13T11:53:42.901 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-09-13T11:53:42.942 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-09-13T11:53:42.942 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-09-13T11:53:43.631 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-09-13T11:53:43.631 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-09-13T11:53:45.171 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-09-13T11:53:45.260 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-13T11:53:45.299 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-13T11:53:45.324 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-13T11:53:45.347 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T11:53:45.374 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T11:53:45.550 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T11:53:45.604 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T11:53:45.992 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-13T11:53:46.092 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:53:46.177 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:53:46.275 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 8/34 2024-09-13T11:53:46.310 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T11:53:46.356 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T11:53:46.393 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-13T11:53:46.438 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 11/34 2024-09-13T11:53:46.508 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T11:53:46.538 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T11:53:46.585 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T11:53:46.620 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-13T11:53:46.740 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-13T11:53:46.779 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-13T11:53:46.820 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-13T11:53:46.852 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-13T11:53:46.881 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-13T11:53:46.915 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-13T11:53:47.253 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-13T11:53:47.293 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T11:53:47.402 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T11:53:47.438 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 23/34 2024-09-13T11:53:47.476 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-13T11:53:48.005 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-13T11:53:48.046 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T11:53:48.101 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T11:53:48.182 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 27/34 2024-09-13T11:53:48.215 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 28/34 2024-09-13T11:53:50.644 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:53:50.686 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:53:51.297 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:53:51.331 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T11:53:51.457 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T11:53:51.457 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T11:53:51.457 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T11:53:51.457 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:53:51.499 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:54:02.092 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:54:02.092 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /sys 2024-09-13T11:54:02.093 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /proc 2024-09-13T11:54:02.093 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /mnt 2024-09-13T11:54:02.093 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /var/tmp 2024-09-13T11:54:02.093 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /home 2024-09-13T11:54:02.093 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /root 2024-09-13T11:54:02.094 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /tmp 2024-09-13T11:54:02.094 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:03.131 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T11:54:03.158 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T11:54:03.158 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:54:03.158 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T11:54:03.158 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:54:03.158 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:54:03.158 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:03.160 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T11:54:03.250 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T11:54:03.250 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:54:04.495 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:54:04.495 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 1/34 2024-09-13T11:54:04.495 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2/34 2024-09-13T11:54:04.495 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 3/34 2024-09-13T11:54:04.495 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 4/34 2024-09-13T11:54:04.495 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 5/34 2024-09-13T11:54:04.496 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 6/34 2024-09-13T11:54:04.496 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:54:04.496 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 8/34 2024-09-13T11:54:04.496 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 9/34 2024-09-13T11:54:04.496 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 10/34 2024-09-13T11:54:04.496 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 11/34 2024-09-13T11:54:04.496 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T11:54:04.496 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 13/34 2024-09-13T11:54:04.496 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-13T11:54:04.496 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-13T11:54:04.496 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-13T11:54:04.496 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-13T11:54:04.496 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-13T11:54:04.497 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-13T11:54:04.497 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-13T11:54:04.497 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-13T11:54:04.497 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-13T11:54:04.497 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-13T11:54:04.497 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-13T11:54:04.497 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-13T11:54:04.497 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-13T11:54:04.497 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-13T11:54:04.498 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-13T11:54:04.498 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-13T11:54:04.499 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-13T11:54:04.499 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:54:04.499 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-13T11:54:04.499 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 33/34 2024-09-13T11:54:05.640 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:54:05.641 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:05.641 INFO:teuthology.orchestra.run.smithi032.stdout:Upgraded: 2024-09-13T11:54:05.641 INFO:teuthology.orchestra.run.smithi032.stdout: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:54:05.641 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:54:05.641 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-09-13T11:54:05.641 INFO:teuthology.orchestra.run.smithi032.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T11:54:05.641 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:54:05.641 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:54:05.641 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:54:05.641 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:54:05.642 INFO:teuthology.orchestra.run.smithi032.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T11:54:05.642 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T11:54:05.642 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T11:54:05.642 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:54:05.642 INFO:teuthology.orchestra.run.smithi032.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T11:54:05.642 INFO:teuthology.orchestra.run.smithi032.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T11:54:05.642 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T11:54:05.642 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:54:05.642 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T11:54:05.642 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:54:05.642 INFO:teuthology.orchestra.run.smithi032.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T11:54:05.642 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T11:54:05.643 INFO:teuthology.orchestra.run.smithi032.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T11:54:05.643 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:54:05.643 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:54:05.643 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:54:05.643 INFO:teuthology.orchestra.run.smithi032.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T11:54:05.643 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:54:05.643 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:54:05.643 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:54:05.643 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-13T11:54:05.643 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-13T11:54:05.643 INFO:teuthology.orchestra.run.smithi032.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-13T11:54:05.643 INFO:teuthology.orchestra.run.smithi032.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T11:54:05.643 INFO:teuthology.orchestra.run.smithi032.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T11:54:05.643 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:05.644 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-13T11:54:05.988 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-test 2024-09-13T11:54:06.574 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:29 ago on Fri 13 Sep 2024 11:53:37 AM UTC. 2024-09-13T11:54:06.722 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-13T11:54:06.723 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:54:06.723 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-09-13T11:54:06.723 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:54:06.723 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-09-13T11:54:06.723 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 50 M 2024-09-13T11:54:06.723 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-09-13T11:54:06.723 INFO:teuthology.orchestra.run.smithi032.stdout: libcephsqlite x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T11:54:06.724 INFO:teuthology.orchestra.run.smithi032.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T11:54:06.724 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T11:54:06.724 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:06.724 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-09-13T11:54:06.724 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:54:06.724 INFO:teuthology.orchestra.run.smithi032.stdout:Install 4 Packages 2024-09-13T11:54:06.724 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:06.725 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 51 M 2024-09-13T11:54:06.725 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 216 M 2024-09-13T11:54:06.725 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-09-13T11:54:06.912 INFO:teuthology.orchestra.run.smithi032.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.4 MB/s | 304 kB 00:00 2024-09-13T11:54:06.954 INFO:teuthology.orchestra.run.smithi032.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-13T11:54:07.013 INFO:teuthology.orchestra.run.smithi032.stdout:(3/4): libcephsqlite-19.3.0-4697.gd58f7306.el9. 714 kB/s | 161 kB 00:00 2024-09-13T11:54:08.380 INFO:teuthology.orchestra.run.smithi032.stdout:(4/4): ceph-test-19.3.0-4697.gd58f7306.el9.x86_ 31 MB/s | 50 MB 00:01 2024-09-13T11:54:08.382 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:54:08.382 INFO:teuthology.orchestra.run.smithi032.stdout:Total 31 MB/s | 51 MB 00:01 2024-09-13T11:54:08.411 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-09-13T11:54:08.436 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-09-13T11:54:08.437 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-09-13T11:54:08.834 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-09-13T11:54:08.834 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-09-13T11:54:09.391 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-09-13T11:54:09.494 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T11:54:09.549 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T11:54:09.573 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T11:54:14.661 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T11:54:14.745 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T11:54:17.395 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T11:54:17.395 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 1/4 2024-09-13T11:54:17.395 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2/4 2024-09-13T11:54:17.395 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T11:54:17.722 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T11:54:17.722 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:17.722 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-09-13T11:54:17.722 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:54:17.722 INFO:teuthology.orchestra.run.smithi032.stdout: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:54:17.722 INFO:teuthology.orchestra.run.smithi032.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T11:54:17.722 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T11:54:17.722 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:17.722 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-13T11:54:17.986 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph 2024-09-13T11:54:18.571 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:41 ago on Fri 13 Sep 2024 11:53:37 AM UTC. 2024-09-13T11:54:18.725 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-13T11:54:18.727 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:54:18.727 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-09-13T11:54:18.727 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:54:18.727 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-09-13T11:54:18.728 INFO:teuthology.orchestra.run.smithi032.stdout: ceph x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 6.6 k 2024-09-13T11:54:18.728 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-09-13T11:54:18.728 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 2.4 M 2024-09-13T11:54:18.728 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 1.5 M 2024-09-13T11:54:18.728 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 291 k 2024-09-13T11:54:18.728 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 4.8 M 2024-09-13T11:54:18.728 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 17 M 2024-09-13T11:54:18.728 INFO:teuthology.orchestra.run.smithi032.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-13T11:54:18.728 INFO:teuthology.orchestra.run.smithi032.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-13T11:54:18.728 INFO:teuthology.orchestra.run.smithi032.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-13T11:54:18.728 INFO:teuthology.orchestra.run.smithi032.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T11:54:18.728 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T11:54:18.728 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T11:54:18.729 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T11:54:18.729 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T11:54:18.729 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T11:54:18.729 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T11:54:18.729 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-collections 2024-09-13T11:54:18.729 INFO:teuthology.orchestra.run.smithi032.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-13T11:54:18.729 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T11:54:18.729 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T11:54:18.729 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T11:54:18.729 INFO:teuthology.orchestra.run.smithi032.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T11:54:18.729 INFO:teuthology.orchestra.run.smithi032.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T11:54:18.729 INFO:teuthology.orchestra.run.smithi032.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T11:54:18.729 INFO:teuthology.orchestra.run.smithi032.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T11:54:18.729 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T11:54:18.730 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T11:54:18.730 INFO:teuthology.orchestra.run.smithi032.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T11:54:18.730 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T11:54:18.730 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T11:54:18.730 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T11:54:18.730 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T11:54:18.730 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T11:54:18.730 INFO:teuthology.orchestra.run.smithi032.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T11:54:18.730 INFO:teuthology.orchestra.run.smithi032.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T11:54:18.730 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T11:54:18.730 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T11:54:18.730 INFO:teuthology.orchestra.run.smithi032.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T11:54:18.730 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:18.731 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-09-13T11:54:18.731 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:54:18.731 INFO:teuthology.orchestra.run.smithi032.stdout:Install 37 Packages 2024-09-13T11:54:18.731 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:18.731 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 31 M 2024-09-13T11:54:18.731 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 109 M 2024-09-13T11:54:18.731 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-09-13T11:54:19.717 INFO:teuthology.orchestra.run.smithi032.stdout:(1/37): ceph-19.3.0-4697.gd58f7306.el9.x86_64.r 52 kB/s | 6.6 kB 00:00 2024-09-13T11:54:19.942 INFO:teuthology.orchestra.run.smithi032.stdout:(2/37): ceph-mgr-19.3.0-4697.gd58f7306.el9.x86_ 4.4 MB/s | 1.5 MB 00:00 2024-09-13T11:54:20.001 INFO:teuthology.orchestra.run.smithi032.stdout:(3/37): ceph-mds-19.3.0-4697.gd58f7306.el9.x86_ 5.8 MB/s | 2.4 MB 00:00 2024-09-13T11:54:20.126 INFO:teuthology.orchestra.run.smithi032.stdout:(4/37): ceph-mon-19.3.0-4697.gd58f7306.el9.x86_ 12 MB/s | 4.8 MB 00:00 2024-09-13T11:54:20.151 INFO:teuthology.orchestra.run.smithi032.stdout:(5/37): ceph-mgr-modules-core-19.3.0-4697.gd58f 1.9 MB/s | 291 kB 00:00 2024-09-13T11:54:20.522 INFO:teuthology.orchestra.run.smithi032.stdout:(6/37): ceph-osd-19.3.0-4697.gd58f7306.el9.x86_ 30 MB/s | 17 MB 00:00 2024-09-13T11:54:20.855 INFO:teuthology.orchestra.run.smithi032.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 347 kB/s | 253 kB 00:00 2024-09-13T11:54:20.964 INFO:teuthology.orchestra.run.smithi032.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 1.2 MB/s | 135 kB 00:00 2024-09-13T11:54:20.997 INFO:teuthology.orchestra.run.smithi032.stdout:(9/37): python3-ply-3.11-14.el9.noarch.rpm 224 kB/s | 106 kB 00:00 2024-09-13T11:54:21.081 INFO:teuthology.orchestra.run.smithi032.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 1.0 MB/s | 125 kB 00:00 2024-09-13T11:54:21.181 INFO:teuthology.orchestra.run.smithi032.stdout:(11/37): python3-cryptography-36.0.1-4.el9.x86_ 1.2 MB/s | 1.2 MB 00:01 2024-09-13T11:54:21.207 INFO:teuthology.orchestra.run.smithi032.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.0 MB/s | 215 kB 00:00 2024-09-13T11:54:21.257 INFO:teuthology.orchestra.run.smithi032.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 188 kB 00:00 2024-09-13T11:54:21.323 INFO:teuthology.orchestra.run.smithi032.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 298 kB/s | 35 kB 00:00 2024-09-13T11:54:21.366 INFO:teuthology.orchestra.run.smithi032.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 711 kB/s | 77 kB 00:00 2024-09-13T11:54:21.424 INFO:teuthology.orchestra.run.smithi032.stdout:(16/37): python3-mako-1.1.4-6.el9.noarch.rpm 710 kB/s | 172 kB 00:00 2024-09-13T11:54:21.475 INFO:teuthology.orchestra.run.smithi032.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 338 kB/s | 51 kB 00:00 2024-09-13T11:54:21.500 INFO:teuthology.orchestra.run.smithi032.stdout:(18/37): lua-devel-5.4.4-4.el9.x86_64.rpm 296 kB/s | 22 kB 00:00 2024-09-13T11:54:21.525 INFO:teuthology.orchestra.run.smithi032.stdout:(19/37): python3-toml-0.10.2-6.el9.noarch.rpm 262 kB/s | 42 kB 00:00 2024-09-13T11:54:21.584 INFO:teuthology.orchestra.run.smithi032.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 521 kB/s | 43 kB 00:00 2024-09-13T11:54:21.609 INFO:teuthology.orchestra.run.smithi032.stdout:(21/37): luarocks-3.9.2-5.el9.noarch.rpm 1.1 MB/s | 151 kB 00:00 2024-09-13T11:54:21.634 INFO:teuthology.orchestra.run.smithi032.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 432 kB/s | 11 kB 00:00 2024-09-13T11:54:21.659 INFO:teuthology.orchestra.run.smithi032.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 710 kB/s | 18 kB 00:00 2024-09-13T11:54:21.685 INFO:teuthology.orchestra.run.smithi032.stdout:(24/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-13T11:54:21.710 INFO:teuthology.orchestra.run.smithi032.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-09-13T11:54:21.737 INFO:teuthology.orchestra.run.smithi032.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-13T11:54:21.760 INFO:teuthology.orchestra.run.smithi032.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-09-13T11:54:21.786 INFO:teuthology.orchestra.run.smithi032.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 922 kB/s | 46 kB 00:00 2024-09-13T11:54:21.819 INFO:teuthology.orchestra.run.smithi032.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-09-13T11:54:21.844 INFO:teuthology.orchestra.run.smithi032.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 942 kB/s | 79 kB 00:00 2024-09-13T11:54:21.862 INFO:teuthology.orchestra.run.smithi032.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 392 kB/s | 16 kB 00:00 2024-09-13T11:54:21.878 INFO:teuthology.orchestra.run.smithi032.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 2.1 MB/s | 36 kB 00:00 2024-09-13T11:54:21.912 INFO:teuthology.orchestra.run.smithi032.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-09-13T11:54:21.941 INFO:teuthology.orchestra.run.smithi032.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-13T11:54:21.963 INFO:teuthology.orchestra.run.smithi032.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 800 kB/s | 20 kB 00:00 2024-09-13T11:54:21.988 INFO:teuthology.orchestra.run.smithi032.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-09-13T11:54:22.021 INFO:teuthology.orchestra.run.smithi032.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-13T11:54:22.025 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:54:22.025 INFO:teuthology.orchestra.run.smithi032.stdout:Total 9.3 MB/s | 31 MB 00:03 2024-09-13T11:54:22.184 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-09-13T11:54:22.229 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-09-13T11:54:22.229 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-09-13T11:54:22.736 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-09-13T11:54:22.737 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-09-13T11:54:24.117 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-09-13T11:54:24.183 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-13T11:54:24.216 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-13T11:54:24.243 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-13T11:54:24.321 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-13T11:54:24.368 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-13T11:54:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-13T11:54:24.498 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-13T11:54:24.582 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-13T11:54:24.637 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-13T11:54:24.672 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-13T11:54:24.701 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-13T11:54:24.736 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-13T11:54:24.769 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-13T11:54:24.810 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-13T11:54:24.853 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-13T11:54:24.895 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-13T11:54:24.945 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-13T11:54:24.984 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-13T11:54:25.061 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-13T11:54:25.216 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-13T11:54:25.295 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-13T11:54:25.346 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-13T11:54:25.391 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-13T11:54:25.450 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-13T11:54:25.522 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-13T11:54:25.720 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-13T11:54:25.770 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-13T11:54:25.852 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-13T11:54:25.950 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-13T11:54:25.986 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-13T11:54:26.125 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-13T11:54:26.299 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 32/37 2024-09-13T11:54:26.329 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T11:54:26.357 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T11:54:26.357 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:54:26.357 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T11:54:26.357 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T11:54:26.357 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T11:54:26.357 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:27.930 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T11:54:27.959 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T11:54:27.959 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:54:27.959 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T11:54:27.959 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T11:54:27.960 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T11:54:27.960 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:28.440 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T11:54:28.465 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T11:54:28.465 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:54:28.465 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T11:54:28.465 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T11:54:28.465 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T11:54:28.465 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:28.698 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T11:54:28.723 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T11:54:28.724 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:54:28.724 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T11:54:28.724 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T11:54:28.724 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T11:54:28.724 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:28.790 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T11:54:30.697 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T11:54:30.698 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 1/37 2024-09-13T11:54:30.698 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2/37 2024-09-13T11:54:30.698 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 3/37 2024-09-13T11:54:30.698 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 4/37 2024-09-13T11:54:30.698 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 5/37 2024-09-13T11:54:30.698 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 6/37 2024-09-13T11:54:30.698 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-13T11:54:30.698 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-13T11:54:30.698 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-13T11:54:30.698 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-13T11:54:30.698 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-13T11:54:30.698 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-13T11:54:30.699 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-13T11:54:30.699 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-13T11:54:30.699 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-13T11:54:30.699 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-13T11:54:30.699 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-13T11:54:30.699 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-13T11:54:30.699 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-13T11:54:30.699 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-13T11:54:30.699 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-13T11:54:30.699 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-13T11:54:30.700 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-13T11:54:30.700 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-13T11:54:30.700 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-13T11:54:30.700 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-13T11:54:30.700 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-13T11:54:30.700 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-13T11:54:30.700 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-13T11:54:30.700 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-13T11:54:30.700 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-13T11:54:30.700 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-13T11:54:30.700 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-13T11:54:30.700 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-13T11:54:30.700 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-13T11:54:30.701 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-13T11:54:31.736 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-13T11:54:31.736 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:31.736 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-09-13T11:54:31.736 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:54:31.736 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:54:31.736 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:54:31.736 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:54:31.737 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:54:31.737 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:54:31.737 INFO:teuthology.orchestra.run.smithi032.stdout: lua-5.4.4-4.el9.x86_64 2024-09-13T11:54:31.737 INFO:teuthology.orchestra.run.smithi032.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-13T11:54:31.737 INFO:teuthology.orchestra.run.smithi032.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-13T11:54:31.737 INFO:teuthology.orchestra.run.smithi032.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T11:54:31.737 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T11:54:31.737 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T11:54:31.737 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T11:54:31.737 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T11:54:31.737 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T11:54:31.737 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T11:54:31.737 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T11:54:31.738 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T11:54:31.738 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T11:54:31.738 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T11:54:31.738 INFO:teuthology.orchestra.run.smithi032.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T11:54:31.738 INFO:teuthology.orchestra.run.smithi032.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T11:54:31.738 INFO:teuthology.orchestra.run.smithi032.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T11:54:31.738 INFO:teuthology.orchestra.run.smithi032.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T11:54:31.738 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T11:54:31.738 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T11:54:31.738 INFO:teuthology.orchestra.run.smithi032.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T11:54:31.738 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T11:54:31.738 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T11:54:31.738 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T11:54:31.739 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T11:54:31.739 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T11:54:31.739 INFO:teuthology.orchestra.run.smithi032.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T11:54:31.739 INFO:teuthology.orchestra.run.smithi032.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T11:54:31.739 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T11:54:31.739 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T11:54:31.739 INFO:teuthology.orchestra.run.smithi032.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T11:54:31.739 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:31.739 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-13T11:54:32.078 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-base 2024-09-13T11:54:32.673 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:55 ago on Fri 13 Sep 2024 11:53:37 AM UTC. 2024-09-13T11:54:32.766 INFO:teuthology.orchestra.run.smithi032.stdout:Package ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:54:32.811 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-13T11:54:32.812 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-09-13T11:54:32.812 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-13T11:54:32.871 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install cephadm 2024-09-13T11:54:33.459 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:56 ago on Fri 13 Sep 2024 11:53:37 AM UTC. 2024-09-13T11:54:33.599 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-13T11:54:33.599 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:54:33.599 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-09-13T11:54:33.599 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:54:33.600 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-09-13T11:54:33.600 INFO:teuthology.orchestra.run.smithi032.stdout: cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 783 k 2024-09-13T11:54:33.600 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:33.600 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-09-13T11:54:33.600 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:54:33.600 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-09-13T11:54:33.600 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:33.600 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 783 k 2024-09-13T11:54:33.600 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 789 k 2024-09-13T11:54:33.600 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-09-13T11:54:33.863 INFO:teuthology.orchestra.run.smithi032.stdout:cephadm-19.3.0-4697.gd58f7306.el9.noarch.rpm 2.9 MB/s | 783 kB 00:00 2024-09-13T11:54:33.863 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:54:33.864 INFO:teuthology.orchestra.run.smithi032.stdout:Total 2.9 MB/s | 783 kB 00:00 2024-09-13T11:54:33.864 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-09-13T11:54:33.870 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-09-13T11:54:33.870 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-09-13T11:54:33.876 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-09-13T11:54:33.876 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-09-13T11:54:33.948 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-09-13T11:54:34.403 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:54:34.503 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:54:34.838 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:54:35.088 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:54:35.088 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:35.088 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-09-13T11:54:35.088 INFO:teuthology.orchestra.run.smithi032.stdout: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:54:35.089 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:35.089 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-13T11:54:35.287 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T11:54:35.885 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:58 ago on Fri 13 Sep 2024 11:53:37 AM UTC. 2024-09-13T11:54:36.024 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-13T11:54:36.025 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:54:36.025 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-09-13T11:54:36.025 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:54:36.025 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-09-13T11:54:36.026 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 145 k 2024-09-13T11:54:36.026 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:36.026 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-09-13T11:54:36.026 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:54:36.026 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-09-13T11:54:36.026 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:36.026 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 145 k 2024-09-13T11:54:36.026 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 432 k 2024-09-13T11:54:36.026 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-09-13T11:54:36.228 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-immutable-object-cache-19.3.0-4697.gd58f73 717 kB/s | 145 kB 00:00 2024-09-13T11:54:36.229 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:54:36.229 INFO:teuthology.orchestra.run.smithi032.stdout:Total 711 kB/s | 145 kB 00:00 2024-09-13T11:54:36.229 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-09-13T11:54:36.238 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-09-13T11:54:36.238 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-09-13T11:54:36.298 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-09-13T11:54:36.298 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-09-13T11:54:36.445 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-09-13T11:54:36.504 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:54:36.532 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:54:36.532 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:54:36.532 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T11:54:36.532 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:37.459 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:54:37.459 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:37.459 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-09-13T11:54:37.459 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:54:37.459 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:37.459 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-13T11:54:37.692 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr 2024-09-13T11:54:38.285 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:01 ago on Fri 13 Sep 2024 11:53:37 AM UTC. 2024-09-13T11:54:38.379 INFO:teuthology.orchestra.run.smithi032.stdout:Package ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:54:38.422 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-13T11:54:38.424 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-09-13T11:54:38.424 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-13T11:54:38.486 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T11:54:39.071 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:02 ago on Fri 13 Sep 2024 11:53:37 AM UTC. 2024-09-13T11:54:39.210 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-13T11:54:39.211 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:54:39.211 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-09-13T11:54:39.212 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:54:39.212 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-09-13T11:54:39.212 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 14 M 2024-09-13T11:54:39.212 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-09-13T11:54:39.212 INFO:teuthology.orchestra.run.smithi032.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-13T11:54:39.212 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 34 k 2024-09-13T11:54:39.212 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 16 k 2024-09-13T11:54:39.212 INFO:teuthology.orchestra.run.smithi032.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-13T11:54:39.212 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-13T11:54:39.212 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-13T11:54:39.212 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-13T11:54:39.212 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-13T11:54:39.213 INFO:teuthology.orchestra.run.smithi032.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-13T11:54:39.213 INFO:teuthology.orchestra.run.smithi032.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T11:54:39.213 INFO:teuthology.orchestra.run.smithi032.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T11:54:39.213 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:39.213 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-09-13T11:54:39.213 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:54:39.213 INFO:teuthology.orchestra.run.smithi032.stdout:Install 12 Packages 2024-09-13T11:54:39.213 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:39.214 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 19 M 2024-09-13T11:54:39.214 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 159 M 2024-09-13T11:54:39.214 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-09-13T11:54:39.766 INFO:teuthology.orchestra.run.smithi032.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4697.gd5 243 kB/s | 34 kB 00:00 2024-09-13T11:54:39.782 INFO:teuthology.orchestra.run.smithi032.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4697.gd58 104 kB/s | 16 kB 00:00 2024-09-13T11:54:39.908 INFO:teuthology.orchestra.run.smithi032.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.1 MB/s | 269 kB 00:00 2024-09-13T11:54:39.958 INFO:teuthology.orchestra.run.smithi032.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.3 MB/s | 1.0 MB 00:00 2024-09-13T11:54:40.017 INFO:teuthology.orchestra.run.smithi032.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.2 MB/s | 548 kB 00:00 2024-09-13T11:54:40.034 INFO:teuthology.orchestra.run.smithi032.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 1.1 MB/s | 19 kB 00:00 2024-09-13T11:54:40.092 INFO:teuthology.orchestra.run.smithi032.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 35 MB/s | 2.0 MB 00:00 2024-09-13T11:54:40.126 INFO:teuthology.orchestra.run.smithi032.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 4.2 MB/s | 144 kB 00:00 2024-09-13T11:54:40.151 INFO:teuthology.orchestra.run.smithi032.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-13T11:54:40.185 INFO:teuthology.orchestra.run.smithi032.stdout:(10/12): protobuf-compiler-3.14.0-13.el9.x86_64 3.0 MB/s | 863 kB 00:00 2024-09-13T11:54:40.218 INFO:teuthology.orchestra.run.smithi032.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-09-13T11:54:40.452 INFO:teuthology.orchestra.run.smithi032.stdout:(12/12): ceph-mgr-dashboard-19.3.0-4697.gd58f73 17 MB/s | 14 MB 00:00 2024-09-13T11:54:40.455 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:54:40.456 INFO:teuthology.orchestra.run.smithi032.stdout:Total 16 MB/s | 19 MB 00:01 2024-09-13T11:54:40.507 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-09-13T11:54:40.528 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-09-13T11:54:40.528 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-09-13T11:54:40.824 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-09-13T11:54:40.825 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-09-13T11:54:41.441 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-09-13T11:54:41.520 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-13T11:54:41.587 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-13T11:54:41.639 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-13T11:54:41.665 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-13T11:54:41.786 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-13T11:54:41.930 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-13T11:54:42.021 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-13T11:54:42.070 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-13T11:54:42.098 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-13T11:54:42.131 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 10/12 2024-09-13T11:54:45.507 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 11/12 2024-09-13T11:54:45.567 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T11:54:47.591 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T11:54:47.591 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 1/12 2024-09-13T11:54:47.591 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 2/12 2024-09-13T11:54:47.591 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 3/12 2024-09-13T11:54:47.591 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-13T11:54:47.591 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-13T11:54:47.591 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-13T11:54:47.591 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-13T11:54:47.591 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-13T11:54:47.592 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-13T11:54:47.592 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-13T11:54:47.592 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-13T11:54:47.911 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-13T11:54:47.911 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:47.911 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-09-13T11:54:47.911 INFO:teuthology.orchestra.run.smithi032.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-13T11:54:47.912 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:54:47.912 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:54:47.912 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:54:47.912 INFO:teuthology.orchestra.run.smithi032.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-13T11:54:47.912 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-13T11:54:47.912 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-13T11:54:47.912 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-13T11:54:47.912 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-13T11:54:47.912 INFO:teuthology.orchestra.run.smithi032.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-13T11:54:47.912 INFO:teuthology.orchestra.run.smithi032.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T11:54:47.912 INFO:teuthology.orchestra.run.smithi032.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T11:54:47.912 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:47.913 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-13T11:54:48.147 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T11:54:48.733 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:11 ago on Fri 13 Sep 2024 11:53:37 AM UTC. 2024-09-13T11:54:48.872 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-13T11:54:48.873 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================= 2024-09-13T11:54:48.873 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-09-13T11:54:48.873 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================= 2024-09-13T11:54:48.873 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-09-13T11:54:48.873 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 7.4 M 2024-09-13T11:54:48.873 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-09-13T11:54:48.873 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T11:54:48.873 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T11:54:48.874 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T11:54:48.874 INFO:teuthology.orchestra.run.smithi032.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T11:54:48.874 INFO:teuthology.orchestra.run.smithi032.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T11:54:48.874 INFO:teuthology.orchestra.run.smithi032.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T11:54:48.874 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T11:54:48.874 INFO:teuthology.orchestra.run.smithi032.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T11:54:48.874 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T11:54:48.874 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T11:54:48.874 INFO:teuthology.orchestra.run.smithi032.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T11:54:48.874 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:48.874 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-09-13T11:54:48.874 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================= 2024-09-13T11:54:48.875 INFO:teuthology.orchestra.run.smithi032.stdout:Install 12 Packages 2024-09-13T11:54:48.875 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:48.875 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 38 M 2024-09-13T11:54:48.875 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 212 M 2024-09-13T11:54:48.875 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-09-13T11:54:49.117 INFO:teuthology.orchestra.run.smithi032.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 960 kB/s | 191 kB 00:00 2024-09-13T11:54:49.200 INFO:teuthology.orchestra.run.smithi032.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 393 kB/s | 33 kB 00:00 2024-09-13T11:54:49.225 INFO:teuthology.orchestra.run.smithi032.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.5 MB/s | 801 kB 00:00 2024-09-13T11:54:49.317 INFO:teuthology.orchestra.run.smithi032.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 199 kB/s | 18 kB 00:00 2024-09-13T11:54:49.376 INFO:teuthology.orchestra.run.smithi032.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 660 kB/s | 38 kB 00:00 2024-09-13T11:54:49.468 INFO:teuthology.orchestra.run.smithi032.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-46 13 MB/s | 7.4 MB 00:00 2024-09-13T11:54:49.535 INFO:teuthology.orchestra.run.smithi032.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.9 MB/s | 3.0 MB 00:00 2024-09-13T11:54:49.643 INFO:teuthology.orchestra.run.smithi032.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 47 MB/s | 5.1 MB 00:00 2024-09-13T11:54:49.719 INFO:teuthology.orchestra.run.smithi032.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 14 MB/s | 4.9 MB 00:00 2024-09-13T11:54:49.744 INFO:teuthology.orchestra.run.smithi032.stdout:(10/12): python3-devel-3.9.19-8.el9.x86_64.rpm 888 kB/s | 245 kB 00:00 2024-09-13T11:54:49.769 INFO:teuthology.orchestra.run.smithi032.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.0 MB/s | 250 kB 00:00 2024-09-13T11:54:50.003 INFO:teuthology.orchestra.run.smithi032.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 57 MB/s | 16 MB 00:00 2024-09-13T11:54:50.004 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:54:50.004 INFO:teuthology.orchestra.run.smithi032.stdout:Total 34 MB/s | 38 MB 00:01 2024-09-13T11:54:50.368 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-09-13T11:54:50.395 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-09-13T11:54:50.395 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-09-13T11:54:50.850 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-09-13T11:54:50.850 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-09-13T11:54:51.632 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-09-13T11:54:51.710 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T11:54:51.759 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T11:54:51.880 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T11:54:51.927 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T11:54:52.120 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T11:54:52.151 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T11:54:52.232 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T11:54:52.954 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T11:54:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T11:54:54.396 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T11:54:55.470 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T11:54:55.541 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T11:54:58.029 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T11:54:58.029 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 1/12 2024-09-13T11:54:58.029 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T11:54:58.029 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T11:54:58.029 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T11:54:58.029 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T11:54:58.029 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T11:54:58.029 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T11:54:58.029 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T11:54:58.029 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T11:54:58.030 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T11:54:58.030 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T11:54:58.471 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T11:54:58.472 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:58.472 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-09-13T11:54:58.472 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:54:58.472 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T11:54:58.472 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T11:54:58.472 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T11:54:58.472 INFO:teuthology.orchestra.run.smithi032.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T11:54:58.472 INFO:teuthology.orchestra.run.smithi032.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T11:54:58.472 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T11:54:58.472 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T11:54:58.472 INFO:teuthology.orchestra.run.smithi032.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T11:54:58.472 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T11:54:58.473 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T11:54:58.473 INFO:teuthology.orchestra.run.smithi032.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T11:54:58.473 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:58.473 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-13T11:54:58.889 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr-rook 2024-09-13T11:54:59.484 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:22 ago on Fri 13 Sep 2024 11:53:37 AM UTC. 2024-09-13T11:54:59.624 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-13T11:54:59.625 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:54:59.625 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-09-13T11:54:59.625 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:54:59.625 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-09-13T11:54:59.625 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-rook noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 49 k 2024-09-13T11:54:59.625 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-09-13T11:54:59.626 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T11:54:59.626 INFO:teuthology.orchestra.run.smithi032.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T11:54:59.626 INFO:teuthology.orchestra.run.smithi032.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T11:54:59.626 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T11:54:59.626 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T11:54:59.626 INFO:teuthology.orchestra.run.smithi032.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T11:54:59.626 INFO:teuthology.orchestra.run.smithi032.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T11:54:59.626 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T11:54:59.626 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T11:54:59.626 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T11:54:59.626 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T11:54:59.626 INFO:teuthology.orchestra.run.smithi032.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T11:54:59.626 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:59.627 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-09-13T11:54:59.627 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:54:59.627 INFO:teuthology.orchestra.run.smithi032.stdout:Install 13 Packages 2024-09-13T11:54:59.627 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:54:59.627 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 2.2 M 2024-09-13T11:54:59.627 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 26 M 2024-09-13T11:54:59.627 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-09-13T11:55:00.091 INFO:teuthology.orchestra.run.smithi032.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 247 kB/s | 19 kB 00:00 2024-09-13T11:55:00.116 INFO:teuthology.orchestra.run.smithi032.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 249 kB/s | 26 kB 00:00 2024-09-13T11:55:00.174 INFO:teuthology.orchestra.run.smithi032.stdout:(3/13): ceph-mgr-rook-19.3.0-4697.gd58f7306.el9 305 kB/s | 49 kB 00:00 2024-09-13T11:55:00.200 INFO:teuthology.orchestra.run.smithi032.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-13T11:55:00.225 INFO:teuthology.orchestra.run.smithi032.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-13T11:55:00.250 INFO:teuthology.orchestra.run.smithi032.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-13T11:55:00.292 INFO:teuthology.orchestra.run.smithi032.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 483 kB/s | 32 kB 00:00 2024-09-13T11:55:00.318 INFO:teuthology.orchestra.run.smithi032.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 211 kB/s | 14 kB 00:00 2024-09-13T11:55:00.343 INFO:teuthology.orchestra.run.smithi032.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.6 MB/s | 279 kB 00:00 2024-09-13T11:55:00.376 INFO:teuthology.orchestra.run.smithi032.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-13T11:55:00.401 INFO:teuthology.orchestra.run.smithi032.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2024-09-13T11:55:00.427 INFO:teuthology.orchestra.run.smithi032.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 705 kB/s | 59 kB 00:00 2024-09-13T11:55:00.468 INFO:teuthology.orchestra.run.smithi032.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-13T11:55:00.471 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:55:00.472 INFO:teuthology.orchestra.run.smithi032.stdout:Total 2.6 MB/s | 2.2 MB 00:00 2024-09-13T11:55:00.541 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-09-13T11:55:00.557 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-09-13T11:55:00.558 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-09-13T11:55:00.655 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-09-13T11:55:00.655 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-09-13T11:55:01.172 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-09-13T11:55:01.354 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T11:55:01.406 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T11:55:01.461 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T11:55:01.496 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T11:55:01.539 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T11:55:01.646 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T11:55:01.755 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T11:55:01.808 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T11:55:02.592 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T11:55:02.645 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T11:55:02.678 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T11:55:02.727 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T11:55:02.756 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T11:55:03.513 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T11:55:03.514 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 1/13 2024-09-13T11:55:03.514 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T11:55:03.514 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T11:55:03.514 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T11:55:03.514 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T11:55:03.514 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T11:55:03.514 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T11:55:03.514 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T11:55:03.514 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T11:55:03.514 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T11:55:03.514 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T11:55:03.515 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T11:55:03.851 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T11:55:03.851 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:03.851 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-09-13T11:55:03.851 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:55:03.851 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T11:55:03.852 INFO:teuthology.orchestra.run.smithi032.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T11:55:03.852 INFO:teuthology.orchestra.run.smithi032.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T11:55:03.852 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T11:55:03.852 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T11:55:03.852 INFO:teuthology.orchestra.run.smithi032.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T11:55:03.852 INFO:teuthology.orchestra.run.smithi032.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T11:55:03.852 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T11:55:03.852 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T11:55:03.852 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T11:55:03.852 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T11:55:03.852 INFO:teuthology.orchestra.run.smithi032.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T11:55:03.852 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:03.853 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-13T11:55:04.085 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T11:55:04.676 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:27 ago on Fri 13 Sep 2024 11:53:37 AM UTC. 2024-09-13T11:55:04.816 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-13T11:55:04.817 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:55:04.817 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-09-13T11:55:04.817 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:55:04.817 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-09-13T11:55:04.817 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 154 k 2024-09-13T11:55:04.817 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-09-13T11:55:04.817 INFO:teuthology.orchestra.run.smithi032.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T11:55:04.817 INFO:teuthology.orchestra.run.smithi032.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T11:55:04.817 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T11:55:04.817 INFO:teuthology.orchestra.run.smithi032.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T11:55:04.817 INFO:teuthology.orchestra.run.smithi032.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T11:55:04.817 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:04.818 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-09-13T11:55:04.818 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:55:04.818 INFO:teuthology.orchestra.run.smithi032.stdout:Install 6 Packages 2024-09-13T11:55:04.818 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:04.818 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 7.0 M 2024-09-13T11:55:04.818 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 33 M 2024-09-13T11:55:04.818 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-09-13T11:55:05.478 INFO:teuthology.orchestra.run.smithi032.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4697.gd58f7306.e 793 kB/s | 154 kB 00:00 2024-09-13T11:55:05.528 INFO:teuthology.orchestra.run.smithi032.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-13T11:55:05.554 INFO:teuthology.orchestra.run.smithi032.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-13T11:55:05.587 INFO:teuthology.orchestra.run.smithi032.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.2 MB/s | 74 kB 00:00 2024-09-13T11:55:05.912 INFO:teuthology.orchestra.run.smithi032.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 395 kB/s | 248 kB 00:00 2024-09-13T11:55:07.646 INFO:teuthology.orchestra.run.smithi032.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 2.5 MB/s | 6.0 MB 00:02 2024-09-13T11:55:07.648 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:55:07.648 INFO:teuthology.orchestra.run.smithi032.stdout:Total 2.5 MB/s | 7.0 MB 00:02 2024-09-13T11:55:07.723 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-09-13T11:55:07.736 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-09-13T11:55:07.736 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-09-13T11:55:07.841 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-09-13T11:55:07.842 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-09-13T11:55:08.169 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-09-13T11:55:08.352 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T11:55:08.407 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T11:55:08.979 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T11:55:09.070 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T11:55:09.146 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T11:55:09.176 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T11:55:09.938 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T11:55:09.938 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/6 2024-09-13T11:55:09.938 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T11:55:09.938 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T11:55:09.938 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T11:55:09.939 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T11:55:10.225 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T11:55:10.225 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:10.225 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-09-13T11:55:10.225 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:55:10.225 INFO:teuthology.orchestra.run.smithi032.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T11:55:10.225 INFO:teuthology.orchestra.run.smithi032.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T11:55:10.225 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T11:55:10.226 INFO:teuthology.orchestra.run.smithi032.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T11:55:10.226 INFO:teuthology.orchestra.run.smithi032.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T11:55:10.226 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:10.226 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-13T11:55:10.477 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-fuse 2024-09-13T11:55:11.069 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:34 ago on Fri 13 Sep 2024 11:53:37 AM UTC. 2024-09-13T11:55:11.208 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-13T11:55:11.209 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:55:11.209 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-09-13T11:55:11.209 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:55:11.209 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-09-13T11:55:11.209 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 900 k 2024-09-13T11:55:11.209 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-09-13T11:55:11.209 INFO:teuthology.orchestra.run.smithi032.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T11:55:11.209 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:11.209 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-09-13T11:55:11.209 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:55:11.209 INFO:teuthology.orchestra.run.smithi032.stdout:Install 2 Packages 2024-09-13T11:55:11.209 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:11.210 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 979 k 2024-09-13T11:55:11.210 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 2.9 M 2024-09-13T11:55:11.210 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-09-13T11:55:11.723 INFO:teuthology.orchestra.run.smithi032.stdout:(1/2): ceph-fuse-19.3.0-4697.gd58f7306.el9.x86_ 3.4 MB/s | 900 kB 00:00 2024-09-13T11:55:11.840 INFO:teuthology.orchestra.run.smithi032.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 212 kB/s | 80 kB 00:00 2024-09-13T11:55:11.840 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:55:11.841 INFO:teuthology.orchestra.run.smithi032.stdout:Total 1.5 MB/s | 979 kB 00:00 2024-09-13T11:55:11.853 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-09-13T11:55:11.877 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-09-13T11:55:11.877 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-09-13T11:55:11.963 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-09-13T11:55:11.964 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-09-13T11:55:12.161 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-09-13T11:55:12.314 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T11:55:12.392 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T11:55:13.102 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T11:55:13.103 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/2 2024-09-13T11:55:13.393 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T11:55:13.393 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:13.393 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-09-13T11:55:13.393 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T11:55:13.393 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:13.393 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-13T11:55:13.672 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-volume 2024-09-13T11:55:14.268 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:37 ago on Fri 13 Sep 2024 11:53:37 AM UTC. 2024-09-13T11:55:14.406 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-13T11:55:14.407 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:55:14.407 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-09-13T11:55:14.407 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:55:14.407 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-09-13T11:55:14.407 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-volume noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 285 k 2024-09-13T11:55:14.407 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:14.407 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-09-13T11:55:14.408 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:55:14.408 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-09-13T11:55:14.408 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:14.408 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 285 k 2024-09-13T11:55:14.408 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 1.3 M 2024-09-13T11:55:14.408 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-09-13T11:55:14.744 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-volume-19.3.0-4697.gd58f7306.el9.noarch.rp 850 kB/s | 285 kB 00:00 2024-09-13T11:55:14.744 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:55:14.745 INFO:teuthology.orchestra.run.smithi032.stdout:Total 845 kB/s | 285 kB 00:00 2024-09-13T11:55:14.745 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-09-13T11:55:14.754 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-09-13T11:55:14.754 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-09-13T11:55:14.789 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-09-13T11:55:14.789 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-09-13T11:55:15.018 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-09-13T11:55:15.079 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:55:15.096 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:55:15.096 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:55:15.096 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-13T11:55:15.097 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:16.088 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:55:16.088 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:16.089 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-09-13T11:55:16.089 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:55:16.089 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:16.089 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-13T11:55:16.399 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install librados-devel 2024-09-13T11:55:16.997 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:39 ago on Fri 13 Sep 2024 11:53:37 AM UTC. 2024-09-13T11:55:17.137 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-13T11:55:17.138 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:55:17.138 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-09-13T11:55:17.138 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:55:17.138 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-09-13T11:55:17.138 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 127 k 2024-09-13T11:55:17.138 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:17.138 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-09-13T11:55:17.138 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:55:17.138 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-09-13T11:55:17.138 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:17.138 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 127 k 2024-09-13T11:55:17.139 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 456 k 2024-09-13T11:55:17.139 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-09-13T11:55:17.323 INFO:teuthology.orchestra.run.smithi032.stdout:librados-devel-19.3.0-4697.gd58f7306.el9.x86_64 686 kB/s | 127 kB 00:00 2024-09-13T11:55:17.324 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:55:17.324 INFO:teuthology.orchestra.run.smithi032.stdout:Total 681 kB/s | 127 kB 00:00 2024-09-13T11:55:17.324 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-09-13T11:55:17.330 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-09-13T11:55:17.330 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-09-13T11:55:17.400 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-09-13T11:55:17.401 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-09-13T11:55:17.562 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-09-13T11:55:17.659 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:55:18.122 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:55:18.375 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:55:18.376 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:18.376 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-09-13T11:55:18.376 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:55:18.376 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:18.376 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-13T11:55:18.684 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install libcephfs2 2024-09-13T11:55:19.294 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:42 ago on Fri 13 Sep 2024 11:53:37 AM UTC. 2024-09-13T11:55:19.388 INFO:teuthology.orchestra.run.smithi032.stdout:Package libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:55:19.435 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-13T11:55:19.436 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-09-13T11:55:19.437 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-13T11:55:19.500 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install libcephfs-devel 2024-09-13T11:55:20.091 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:43 ago on Fri 13 Sep 2024 11:53:37 AM UTC. 2024-09-13T11:55:20.232 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-13T11:55:20.233 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:55:20.233 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-09-13T11:55:20.233 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:55:20.233 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-09-13T11:55:20.233 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 32 k 2024-09-13T11:55:20.233 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:20.233 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-09-13T11:55:20.233 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:55:20.233 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-09-13T11:55:20.233 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:20.234 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 32 k 2024-09-13T11:55:20.234 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 142 k 2024-09-13T11:55:20.234 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-09-13T11:55:20.393 INFO:teuthology.orchestra.run.smithi032.stdout:libcephfs-devel-19.3.0-4697.gd58f7306.el9.x86_6 199 kB/s | 32 kB 00:00 2024-09-13T11:55:20.394 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:55:20.394 INFO:teuthology.orchestra.run.smithi032.stdout:Total 197 kB/s | 32 kB 00:00 2024-09-13T11:55:20.394 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-09-13T11:55:20.398 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-09-13T11:55:20.398 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-09-13T11:55:20.416 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-09-13T11:55:20.416 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-09-13T11:55:20.512 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-09-13T11:55:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:55:20.988 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:55:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:55:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:21.215 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-09-13T11:55:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:55:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:21.215 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-13T11:55:21.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install librados2 2024-09-13T11:55:21.996 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:44 ago on Fri 13 Sep 2024 11:53:37 AM UTC. 2024-09-13T11:55:22.090 INFO:teuthology.orchestra.run.smithi032.stdout:Package librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:55:22.135 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-13T11:55:22.136 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-09-13T11:55:22.136 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-13T11:55:22.195 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install librbd1 2024-09-13T11:55:22.785 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:45 ago on Fri 13 Sep 2024 11:53:37 AM UTC. 2024-09-13T11:55:22.880 INFO:teuthology.orchestra.run.smithi032.stdout:Package librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:55:22.925 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-13T11:55:22.926 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-09-13T11:55:22.926 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-13T11:55:22.990 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-rados 2024-09-13T11:55:23.581 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:46 ago on Fri 13 Sep 2024 11:53:37 AM UTC. 2024-09-13T11:55:23.673 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:55:23.721 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-13T11:55:23.721 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-09-13T11:55:23.722 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-13T11:55:23.777 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-rgw 2024-09-13T11:55:24.367 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:47 ago on Fri 13 Sep 2024 11:53:37 AM UTC. 2024-09-13T11:55:24.460 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:55:24.505 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-13T11:55:24.506 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-09-13T11:55:24.506 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-13T11:55:24.570 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-cephfs 2024-09-13T11:55:25.161 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:48 ago on Fri 13 Sep 2024 11:53:37 AM UTC. 2024-09-13T11:55:25.255 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:55:25.300 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-13T11:55:25.301 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-09-13T11:55:25.301 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-13T11:55:25.375 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-rbd 2024-09-13T11:55:25.968 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:48 ago on Fri 13 Sep 2024 11:53:37 AM UTC. 2024-09-13T11:55:26.059 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:55:26.105 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-13T11:55:26.106 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-09-13T11:55:26.106 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-13T11:55:26.161 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install rbd-fuse 2024-09-13T11:55:26.757 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:49 ago on Fri 13 Sep 2024 11:53:37 AM UTC. 2024-09-13T11:55:26.895 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-13T11:55:26.895 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:55:26.895 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-09-13T11:55:26.895 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:55:26.896 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-09-13T11:55:26.896 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 85 k 2024-09-13T11:55:26.896 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:26.896 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-09-13T11:55:26.896 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:55:26.896 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-09-13T11:55:26.896 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:26.897 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 85 k 2024-09-13T11:55:26.897 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 231 k 2024-09-13T11:55:26.897 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-09-13T11:55:27.205 INFO:teuthology.orchestra.run.smithi032.stdout:rbd-fuse-19.3.0-4697.gd58f7306.el9.x86_64.rpm 277 kB/s | 85 kB 00:00 2024-09-13T11:55:27.206 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:55:27.206 INFO:teuthology.orchestra.run.smithi032.stdout:Total 276 kB/s | 85 kB 00:00 2024-09-13T11:55:27.206 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-09-13T11:55:27.213 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-09-13T11:55:27.213 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-09-13T11:55:27.281 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-09-13T11:55:27.282 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-09-13T11:55:27.437 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-09-13T11:55:27.557 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:55:27.906 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:55:28.182 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:55:28.182 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:28.182 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-09-13T11:55:28.182 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:55:28.182 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:28.182 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-13T11:55:28.434 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install rbd-mirror 2024-09-13T11:55:29.023 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:52 ago on Fri 13 Sep 2024 11:53:37 AM UTC. 2024-09-13T11:55:29.168 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-13T11:55:29.168 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:55:29.168 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-09-13T11:55:29.169 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:55:29.169 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-09-13T11:55:29.169 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T11:55:29.169 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:29.169 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-09-13T11:55:29.169 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:55:29.169 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-09-13T11:55:29.169 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:29.170 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 3.2 M 2024-09-13T11:55:29.170 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 14 M 2024-09-13T11:55:29.170 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-09-13T11:55:29.558 INFO:teuthology.orchestra.run.smithi032.stdout:rbd-mirror-19.3.0-4697.gd58f7306.el9.x86_64.rpm 8.2 MB/s | 3.2 MB 00:00 2024-09-13T11:55:29.558 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:55:29.559 INFO:teuthology.orchestra.run.smithi032.stdout:Total 8.1 MB/s | 3.2 MB 00:00 2024-09-13T11:55:29.559 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-09-13T11:55:29.569 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-09-13T11:55:29.569 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-09-13T11:55:29.661 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-09-13T11:55:29.661 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-09-13T11:55:30.143 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-09-13T11:55:30.200 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:55:30.226 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:55:30.227 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:55:30.227 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T11:55:30.227 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T11:55:30.227 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T11:55:30.227 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:31.324 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:55:31.324 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:31.324 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-09-13T11:55:31.324 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:55:31.324 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:31.325 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-13T11:55:31.562 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install rbd-nbd 2024-09-13T11:55:32.162 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:55 ago on Fri 13 Sep 2024 11:53:37 AM UTC. 2024-09-13T11:55:32.304 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-13T11:55:32.305 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:55:32.305 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-09-13T11:55:32.305 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:55:32.305 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-09-13T11:55:32.305 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 170 k 2024-09-13T11:55:32.305 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:32.305 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-09-13T11:55:32.305 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-13T11:55:32.305 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-09-13T11:55:32.305 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:32.305 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 170 k 2024-09-13T11:55:32.306 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 490 k 2024-09-13T11:55:32.306 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-09-13T11:55:32.519 INFO:teuthology.orchestra.run.smithi032.stdout:rbd-nbd-19.3.0-4697.gd58f7306.el9.x86_64.rpm 793 kB/s | 170 kB 00:00 2024-09-13T11:55:32.519 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:55:32.520 INFO:teuthology.orchestra.run.smithi032.stdout:Total 787 kB/s | 170 kB 00:00 2024-09-13T11:55:32.520 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-09-13T11:55:32.528 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-09-13T11:55:32.528 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-09-13T11:55:32.596 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-09-13T11:55:32.596 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-09-13T11:55:32.761 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-09-13T11:55:32.861 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:55:33.261 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:55:33.528 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:55:33.528 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:33.528 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-09-13T11:55:33.528 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:55:33.528 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-13T11:55:33.528 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-13T11:55:33.766 DEBUG:teuthology.parallel:result is None 2024-09-13T11:55:33.767 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:55:33.919 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T11:55:33.941 INFO:teuthology.orchestra.run.smithi032.stdout:19.3.0-4697.gd58f7306.el9 2024-09-13T11:55:33.942 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4697.gd58f7306.el9 2024-09-13T11:55:33.942 INFO:teuthology.task.install:The correct ceph version 19.3.0-4697.gd58f7306 is installed. 2024-09-13T11:55:33.943 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-13T11:55:33.944 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-09-13T11:55:33.944 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T11:55:34.008 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-13T11:55:34.008 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-09-13T11:55:34.008 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T11:55:34.073 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T11:55:34.138 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-13T11:55:34.139 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-09-13T11:55:34.139 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T11:55:34.204 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T11:55:34.270 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-13T11:55:34.270 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-09-13T11:55:34.271 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T11:55:34.336 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T11:55:34.403 INFO:teuthology.run_tasks:Running task workunit... 2024-09-13T11:55:34.418 INFO:tasks.workunit:Pulling workunits from ref d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:55:34.418 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-13T11:55:34.418 INFO:tasks.workunit:timeout=3h 2024-09-13T11:55:34.419 INFO:tasks.workunit:cleanup=True 2024-09-13T11:55:34.419 DEBUG:teuthology.orchestra.run.smithi032:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-13T11:55:34.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:55:34.459 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-13T11:55:34.459 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-13T11:55:34.514 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-13T11:55:34.515 DEBUG:teuthology.orchestra.run.smithi032:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-13T11:55:34.571 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:55:34.628 INFO:tasks.workunit.client.0.smithi032.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-13T11:56:33.867 INFO:tasks.workunit.client.0.smithi032.stderr:Updating files: 92% (11724/12697) Updating files: 93% (11809/12697) Updating files: 94% (11936/12697) Updating files: 95% (12063/12697) Updating files: 96% (12190/12697) Updating files: 97% (12317/12697) Updating files: 98% (12444/12697) Updating files: 99% (12571/12697) Updating files: 100% (12697/12697) Updating files: 100% (12697/12697), done. 2024-09-13T11:56:34.737 INFO:tasks.workunit.client.0.smithi032.stderr:Note: switching to 'd58f73069a8157a9ace12a0247ee3e8497707ea9'. 2024-09-13T11:56:34.737 INFO:tasks.workunit.client.0.smithi032.stderr: 2024-09-13T11:56:34.737 INFO:tasks.workunit.client.0.smithi032.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-13T11:56:34.737 INFO:tasks.workunit.client.0.smithi032.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-13T11:56:34.737 INFO:tasks.workunit.client.0.smithi032.stderr:state without impacting any branches by switching back to a branch. 2024-09-13T11:56:34.738 INFO:tasks.workunit.client.0.smithi032.stderr: 2024-09-13T11:56:34.738 INFO:tasks.workunit.client.0.smithi032.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-13T11:56:34.738 INFO:tasks.workunit.client.0.smithi032.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-13T11:56:34.738 INFO:tasks.workunit.client.0.smithi032.stderr: 2024-09-13T11:56:34.738 INFO:tasks.workunit.client.0.smithi032.stderr: git switch -c 2024-09-13T11:56:34.738 INFO:tasks.workunit.client.0.smithi032.stderr: 2024-09-13T11:56:34.738 INFO:tasks.workunit.client.0.smithi032.stderr:Or undo this operation with: 2024-09-13T11:56:34.738 INFO:tasks.workunit.client.0.smithi032.stderr: 2024-09-13T11:56:34.738 INFO:tasks.workunit.client.0.smithi032.stderr: git switch - 2024-09-13T11:56:34.738 INFO:tasks.workunit.client.0.smithi032.stderr: 2024-09-13T11:56:34.738 INFO:tasks.workunit.client.0.smithi032.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-13T11:56:34.738 INFO:tasks.workunit.client.0.smithi032.stderr: 2024-09-13T11:56:34.738 INFO:tasks.workunit.client.0.smithi032.stderr:HEAD is now at d58f73069a8 tools/ceph-bluestore-tool: Added zap-device documentation 2024-09-13T11:56:34.742 DEBUG:teuthology.orchestra.run.smithi032:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-13T11:56:34.799 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-09-13T11:56:34.799 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-13T11:56:34.855 INFO:tasks.workunit:Running workunits matching osd on client.0... 2024-09-13T11:56:34.856 INFO:tasks.workunit:Running workunit osd/bad-inc-map.sh... 2024-09-13T11:56:34.856 DEBUG:teuthology.orchestra.run.smithi032:workunit test osd/bad-inc-map.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d58f73069a8157a9ace12a0247ee3e8497707ea9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh 2024-09-13T11:56:34.982 INFO:tasks.workunit.client.0.smithi032.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-13T11:56:35.012 INFO:tasks.workunit.client.0.smithi032.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-13T11:56:35.012 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-13T11:56:35.012 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-13T11:56:35.012 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-13T11:56:35.012 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-13T11:56:35.012 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-13T11:56:35.012 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-13T11:56:35.012 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-13T11:56:35.012 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-13T11:56:35.013 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/bad-inc-map 2024-09-13T11:56:35.013 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:8: run: local dir=td/bad-inc-map 2024-09-13T11:56:35.013 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:9: run: shift 2024-09-13T11:56:35.013 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: export CEPH_MON=127.0.0.1:41173 2024-09-13T11:56:35.013 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: CEPH_MON=127.0.0.1:41173 2024-09-13T11:56:35.013 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:12: run: export CEPH_ARGS 2024-09-13T11:56:35.013 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: uuidgen 2024-09-13T11:56:35.018 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: CEPH_ARGS+='--fsid=e51e787f-f85e-4d21-9be8-7d13be4336e6 --auth-supported=none ' 2024-09-13T11:56:35.018 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:41173 ' 2024-09-13T11:56:35.018 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:15: run: set -e 2024-09-13T11:56:35.019 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: set 2024-09-13T11:56:35.019 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-13T11:56:35.021 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: local funcs=TEST_bad_inc_map 2024-09-13T11:56:35.021 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:18: run: for func in $funcs 2024-09-13T11:56:35.021 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:19: run: setup td/bad-inc-map 2024-09-13T11:56:35.021 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/bad-inc-map 2024-09-13T11:56:35.021 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/bad-inc-map 2024-09-13T11:56:35.021 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/bad-inc-map 2024-09-13T11:56:35.022 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T11:56:35.022 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/bad-inc-map KILL 2024-09-13T11:56:35.022 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T11:56:35.022 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T11:56:35.022 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T11:56:35.022 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T11:56:35.022 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T11:56:35.023 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T11:56:35.023 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T11:56:35.023 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T11:56:35.023 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T11:56:35.024 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T11:56:35.024 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T11:56:35.025 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T11:56:35.026 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:56:35.026 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T11:56:35.026 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T11:56:35.026 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:56:35.026 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T11:56:35.028 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T11:56:35.028 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/bad-inc-map 2024-09-13T11:56:35.029 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T11:56:35.029 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T11:56:35.029 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27402 2024-09-13T11:56:35.029 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27402 2024-09-13T11:56:35.030 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T11:56:35.030 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T11:56:35.030 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/bad-inc-map 2024-09-13T11:56:35.031 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T11:56:35.031 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T11:56:35.031 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27402 2024-09-13T11:56:35.031 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27402 2024-09-13T11:56:35.033 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T11:56:35.033 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-13T11:56:35.034 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-13T11:56:35.034 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T11:56:35.034 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/bad-inc-map 1' TERM HUP INT 2024-09-13T11:56:35.034 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:20: run: TEST_bad_inc_map td/bad-inc-map 2024-09-13T11:56:35.034 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:26: TEST_bad_inc_map: local dir=td/bad-inc-map 2024-09-13T11:56:35.034 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:28: TEST_bad_inc_map: run_mon td/bad-inc-map a 2024-09-13T11:56:35.034 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/bad-inc-map 2024-09-13T11:56:35.034 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T11:56:35.034 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T11:56:35.034 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T11:56:35.034 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/bad-inc-map/a 2024-09-13T11:56:35.034 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/bad-inc-map/a --run-dir=td/bad-inc-map 2024-09-13T11:56:35.468 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T11:56:35.468 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T11:56:35.468 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T11:56:35.468 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T11:56:35.468 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T11:56:35.468 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27402 2024-09-13T11:56:35.468 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27402/$cluster-$name.asok' 2024-09-13T11:56:35.468 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/bad-inc-map/a '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.27402/$cluster-$name.asok' --mon-cluster-log-file=td/bad-inc-map/log --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-13T11:56:35.661 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T11:56:35.662 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T11:56:35.662 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T11:56:35.662 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T11:56:35.662 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T11:56:35.662 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T11:56:35.662 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T11:56:35.662 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T11:56:35.662 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T11:56:35.663 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T11:56:35.663 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T11:56:35.663 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27402 2024-09-13T11:56:35.663 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27402/ceph-mon.a.asok 2024-09-13T11:56:35.663 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T11:56:35.663 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27402/ceph-mon.a.asok config get fsid 2024-09-13T11:56:35.768 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T11:56:35.768 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T11:56:35.768 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T11:56:35.768 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T11:56:35.768 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T11:56:35.768 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T11:56:35.768 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T11:56:35.769 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T11:56:35.769 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T11:56:35.769 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T11:56:35.769 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27402 2024-09-13T11:56:35.769 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27402/ceph-mon.a.asok 2024-09-13T11:56:35.770 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T11:56:35.770 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27402/ceph-mon.a.asok config get mon_host 2024-09-13T11:56:35.849 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:29: TEST_bad_inc_map: run_mgr td/bad-inc-map x 2024-09-13T11:56:35.849 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/bad-inc-map 2024-09-13T11:56:35.850 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T11:56:35.850 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T11:56:35.850 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T11:56:35.850 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/bad-inc-map/x 2024-09-13T11:56:35.850 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T11:56:36.701 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T11:56:36.701 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T11:56:36.701 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T11:56:36.701 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T11:56:36.701 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T11:56:36.701 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27402 2024-09-13T11:56:36.701 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27402/$cluster-$name.asok' 2024-09-13T11:56:36.702 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T11:56:36.702 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/bad-inc-map/x '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.27402/$cluster-$name.asok' --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T11:56:36.732 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:30: TEST_bad_inc_map: run_osd td/bad-inc-map 0 2024-09-13T11:56:36.732 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/bad-inc-map 2024-09-13T11:56:36.732 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T11:56:36.732 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T11:56:36.732 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T11:56:36.732 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/bad-inc-map/0 2024-09-13T11:56:36.732 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e51e787f-f85e-4d21-9be8-7d13be4336e6 --auth-supported=none --mon-host=127.0.0.1:41173 ' 2024-09-13T11:56:36.732 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T11:56:36.733 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T11:56:36.733 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T11:56:36.733 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/0' 2024-09-13T11:56:36.733 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/0/journal' 2024-09-13T11:56:36.733 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T11:56:36.733 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T11:56:36.733 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-09-13T11:56:36.733 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T11:56:36.733 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T11:56:36.733 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T11:56:36.733 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T11:56:36.733 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T11:56:36.734 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27402 2024-09-13T11:56:36.734 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27402/$cluster-$name.asok' 2024-09-13T11:56:36.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27402/$cluster-$name.asok' 2024-09-13T11:56:36.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T11:56:36.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T11:56:36.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T11:56:36.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-09-13T11:56:36.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-09-13T11:56:36.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T11:56:36.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T11:56:36.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T11:56:36.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T11:56:36.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T11:56:36.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T11:56:36.735 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/bad-inc-map/0 2024-09-13T11:56:36.735 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T11:56:36.735 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0651a177-a4aa-4fba-8e0b-e3c6498b30e7 2024-09-13T11:56:36.735 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 0651a177-a4aa-4fba-8e0b-e3c6498b30e7' 2024-09-13T11:56:36.735 INFO:tasks.workunit.client.0.smithi032.stdout:add osd0 0651a177-a4aa-4fba-8e0b-e3c6498b30e7 2024-09-13T11:56:36.736 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T11:56:36.750 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB0KORm0oejLBAAQ7twrQ7Rq1njeWcIPqRMpA== 2024-09-13T11:56:36.750 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB0KORm0oejLBAAQ7twrQ7Rq1njeWcIPqRMpA=="}' 2024-09-13T11:56:36.750 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0651a177-a4aa-4fba-8e0b-e3c6498b30e7 -i td/bad-inc-map/0/new.json 2024-09-13T11:56:37.078 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T11:56:37.088 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/bad-inc-map/0/new.json 2024-09-13T11:56:37.089 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e51e787f-f85e-4d21-9be8-7d13be4336e6 --auth-supported=none --mon-host=127.0.0.1:41173 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.27402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB0KORm0oejLBAAQ7twrQ7Rq1njeWcIPqRMpA== --osd-uuid 0651a177-a4aa-4fba-8e0b-e3c6498b30e7 2024-09-13T11:56:37.142 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:56:37.141+0000 7f9aaa25e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:56:37.147 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:56:37.146+0000 7f9aaa25e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:56:37.148 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:56:37.147+0000 7f9aaa25e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:56:37.149 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:56:37.148+0000 7f9aaa25e640 -1 bdev(0x55872fcd7c00 td/bad-inc-map/0/block) open stat got: (1) Operation not permitted 2024-09-13T11:56:37.149 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:56:37.148+0000 7f9aaa25e640 -1 bluestore(td/bad-inc-map/0) _read_fsid unparsable uuid 2024-09-13T11:56:42.803 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/bad-inc-map/0/keyring 2024-09-13T11:56:42.804 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T11:56:42.804 INFO:tasks.workunit.client.0.smithi032.stdout:adding osd0 key to auth repository 2024-09-13T11:56:42.805 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T11:56:42.805 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/bad-inc-map/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T11:56:43.361 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.0 2024-09-13T11:56:43.362 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T11:56:43.362 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e51e787f-f85e-4d21-9be8-7d13be4336e6 --auth-supported=none --mon-host=127.0.0.1:41173 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.27402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T11:56:43.362 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T11:56:43.362 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T11:56:43.362 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T11:56:43.387 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:56:43.385+0000 7f09dea2f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:56:43.389 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:56:43.387+0000 7f09dea2f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:56:43.390 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:56:43.388+0000 7f09dea2f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:56:43.794 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T11:56:43.795 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T11:56:43.795 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T11:56:43.795 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T11:56:43.795 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T11:56:43.795 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:56:43.795 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T11:56:43.795 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T11:56:43.796 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:56:43.796 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:56:44.221 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:56:44.737 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:56:44.735+0000 7f09dea2f640 -1 Falling back to public interface 2024-09-13T11:56:45.222 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:56:45.222 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:56:45.222 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T11:56:45.222 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T11:56:45.223 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:56:45.223 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:56:45.634 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:56:46.635 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:56:46.635 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:56:46.635 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T11:56:46.636 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T11:56:46.636 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:56:46.636 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:56:47.082 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:56:47.694 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:56:47.692+0000 7f09dea2f640 -1 osd.0 0 log_to_monitors true 2024-09-13T11:56:48.084 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:56:48.084 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:56:48.084 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T11:56:48.084 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T11:56:48.085 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:56:48.085 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:56:48.753 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:56:49.754 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:56:49.754 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:56:49.754 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T11:56:49.755 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T11:56:49.755 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:56:49.755 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:56:50.175 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:56:51.177 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:56:51.177 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:56:51.177 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T11:56:51.177 INFO:tasks.workunit.client.0.smithi032.stdout:5 2024-09-13T11:56:51.178 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:56:51.178 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:56:51.633 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:56:52.634 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:56:52.635 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:56:52.635 INFO:tasks.workunit.client.0.smithi032.stdout:6 2024-09-13T11:56:52.635 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T11:56:52.635 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:56:52.635 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:56:53.052 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:56:54.053 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:56:54.054 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:56:54.054 INFO:tasks.workunit.client.0.smithi032.stdout:7 2024-09-13T11:56:54.054 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T11:56:54.055 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:56:54.055 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:56:54.570 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:56:55.572 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:56:55.572 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:56:55.572 INFO:tasks.workunit.client.0.smithi032.stdout:8 2024-09-13T11:56:55.573 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T11:56:55.573 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:56:55.573 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:56:55.993 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:56:56.995 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:56:56.995 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:56:56.995 INFO:tasks.workunit.client.0.smithi032.stdout:9 2024-09-13T11:56:56.996 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T11:56:56.996 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:56:56.996 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:56:57.545 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:56:58.546 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:56:58.546 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:56:58.546 INFO:tasks.workunit.client.0.smithi032.stdout:10 2024-09-13T11:56:58.547 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T11:56:58.547 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:56:58.547 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:56:59.072 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:57:00.073 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:57:00.073 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:57:00.073 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T11:57:00.074 INFO:tasks.workunit.client.0.smithi032.stdout:11 2024-09-13T11:57:00.074 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:57:00.074 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:57:00.607 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:57:01.609 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:57:01.609 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:57:01.610 INFO:tasks.workunit.client.0.smithi032.stdout:12 2024-09-13T11:57:01.610 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T11:57:01.610 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:57:01.611 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:57:02.138 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:57:03.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:57:03.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:57:03.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T11:57:03.140 INFO:tasks.workunit.client.0.smithi032.stdout:13 2024-09-13T11:57:03.140 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:57:03.140 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:57:04.108 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:57:05.109 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:57:05.109 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:57:05.110 INFO:tasks.workunit.client.0.smithi032.stdout:14 2024-09-13T11:57:05.110 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T11:57:05.110 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:57:05.110 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:57:06.110 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:57:07.112 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:57:07.157 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:57:07.157 INFO:tasks.workunit.client.0.smithi032.stdout:15 2024-09-13T11:57:07.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T11:57:07.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:57:07.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:57:08.123 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:57:09.124 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:57:09.124 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:57:09.124 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T11:57:09.125 INFO:tasks.workunit.client.0.smithi032.stdout:16 2024-09-13T11:57:09.125 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:57:09.125 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:57:09.790 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:57:10.792 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:57:10.792 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:57:10.792 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T11:57:10.792 INFO:tasks.workunit.client.0.smithi032.stdout:17 2024-09-13T11:57:10.793 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:57:10.793 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:57:11.693 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:57:12.694 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:57:12.694 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:57:12.694 INFO:tasks.workunit.client.0.smithi032.stdout:18 2024-09-13T11:57:12.695 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T11:57:12.695 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:57:12.695 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:57:13.132 INFO:tasks.workunit.client.0.smithi032.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2540798877,v1:127.0.0.1:6803/2540798877] [v2:127.0.0.1:6804/2540798877,v1:127.0.0.1:6805/2540798877] exists,up 0651a177-a4aa-4fba-8e0b-e3c6498b30e7 2024-09-13T11:57:13.136 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T11:57:13.136 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T11:57:13.136 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T11:57:13.136 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:31: TEST_bad_inc_map: run_osd td/bad-inc-map 1 2024-09-13T11:57:13.136 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/bad-inc-map 2024-09-13T11:57:13.137 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T11:57:13.137 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T11:57:13.137 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T11:57:13.137 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/bad-inc-map/1 2024-09-13T11:57:13.137 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e51e787f-f85e-4d21-9be8-7d13be4336e6 --auth-supported=none --mon-host=127.0.0.1:41173 ' 2024-09-13T11:57:13.137 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T11:57:13.137 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T11:57:13.137 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T11:57:13.137 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/1' 2024-09-13T11:57:13.137 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/1/journal' 2024-09-13T11:57:13.138 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T11:57:13.138 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T11:57:13.138 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-09-13T11:57:13.138 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T11:57:13.138 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T11:57:13.138 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T11:57:13.138 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T11:57:13.138 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T11:57:13.138 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27402 2024-09-13T11:57:13.138 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27402/$cluster-$name.asok' 2024-09-13T11:57:13.138 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27402/$cluster-$name.asok' 2024-09-13T11:57:13.138 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T11:57:13.138 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T11:57:13.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T11:57:13.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-09-13T11:57:13.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-09-13T11:57:13.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T11:57:13.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T11:57:13.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T11:57:13.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T11:57:13.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T11:57:13.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T11:57:13.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/bad-inc-map/1 2024-09-13T11:57:13.139 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T11:57:13.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d577bf33-7894-4c3b-805e-9d8bc779f2a5 2024-09-13T11:57:13.140 INFO:tasks.workunit.client.0.smithi032.stdout:add osd1 d577bf33-7894-4c3b-805e-9d8bc779f2a5 2024-09-13T11:57:13.140 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 d577bf33-7894-4c3b-805e-9d8bc779f2a5' 2024-09-13T11:57:13.140 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T11:57:13.152 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCZKORmWZkICRAAsLQG/AieydbhO0llp/KIsw== 2024-09-13T11:57:13.153 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCZKORmWZkICRAAsLQG/AieydbhO0llp/KIsw=="}' 2024-09-13T11:57:13.153 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d577bf33-7894-4c3b-805e-9d8bc779f2a5 -i td/bad-inc-map/1/new.json 2024-09-13T11:57:13.638 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T11:57:13.649 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/bad-inc-map/1/new.json 2024-09-13T11:57:13.650 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e51e787f-f85e-4d21-9be8-7d13be4336e6 --auth-supported=none --mon-host=127.0.0.1:41173 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.27402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCZKORmWZkICRAAsLQG/AieydbhO0llp/KIsw== --osd-uuid d577bf33-7894-4c3b-805e-9d8bc779f2a5 2024-09-13T11:57:13.677 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:57:13.676+0000 7fbe956dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:57:13.694 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:57:13.693+0000 7fbe956dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:57:13.695 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:57:13.694+0000 7fbe956dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:57:13.695 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:57:13.694+0000 7fbe956dc640 -1 bdev(0x55d84e1b7c00 td/bad-inc-map/1/block) open stat got: (1) Operation not permitted 2024-09-13T11:57:13.695 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:57:13.694+0000 7fbe956dc640 -1 bluestore(td/bad-inc-map/1) _read_fsid unparsable uuid 2024-09-13T11:57:19.563 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/bad-inc-map/1/keyring 2024-09-13T11:57:19.564 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T11:57:19.565 INFO:tasks.workunit.client.0.smithi032.stdout:adding osd1 key to auth repository 2024-09-13T11:57:19.565 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T11:57:19.565 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/bad-inc-map/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T11:57:20.137 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.1 2024-09-13T11:57:20.138 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T11:57:20.138 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e51e787f-f85e-4d21-9be8-7d13be4336e6 --auth-supported=none --mon-host=127.0.0.1:41173 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.27402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T11:57:20.138 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T11:57:20.138 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T11:57:20.138 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T11:57:20.163 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:57:20.162+0000 7fb313fd4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:57:20.165 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:57:20.164+0000 7fb313fd4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:57:20.166 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:57:20.165+0000 7fb313fd4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:57:20.578 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T11:57:20.578 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T11:57:20.578 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T11:57:20.578 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T11:57:20.578 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T11:57:20.579 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T11:57:20.579 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:57:20.579 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T11:57:20.579 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:57:20.579 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:57:21.013 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:57:21.012+0000 7fb313fd4640 -1 Falling back to public interface 2024-09-13T11:57:21.029 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:57:22.031 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:57:22.031 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:57:22.031 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T11:57:22.032 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T11:57:22.032 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:57:22.032 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:57:22.480 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:57:23.481 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:57:23.481 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:57:23.482 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T11:57:23.482 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T11:57:23.482 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:57:23.482 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:57:23.642 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:57:23.641+0000 7fb313fd4640 -1 osd.1 0 log_to_monitors true 2024-09-13T11:57:23.948 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:57:24.950 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:57:24.950 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:57:24.950 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T11:57:24.950 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T11:57:24.951 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:57:24.951 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:57:25.459 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:57:26.460 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:57:26.460 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:57:26.460 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T11:57:26.461 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T11:57:26.461 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:57:26.461 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:57:26.896 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:57:27.897 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:57:27.898 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:57:27.898 INFO:tasks.workunit.client.0.smithi032.stdout:5 2024-09-13T11:57:27.899 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T11:57:27.899 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:57:27.899 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:57:28.415 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:57:29.417 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:57:29.417 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:57:29.418 INFO:tasks.workunit.client.0.smithi032.stdout:6 2024-09-13T11:57:29.419 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T11:57:29.419 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:57:29.419 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:57:29.870 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:57:30.872 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:57:30.872 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:57:30.873 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T11:57:30.873 INFO:tasks.workunit.client.0.smithi032.stdout:7 2024-09-13T11:57:30.874 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:57:30.874 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:57:31.331 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:57:32.333 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:57:32.333 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:57:32.333 INFO:tasks.workunit.client.0.smithi032.stdout:8 2024-09-13T11:57:32.334 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T11:57:32.334 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:57:32.334 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:57:32.770 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:57:33.772 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:57:33.772 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:57:33.772 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T11:57:33.772 INFO:tasks.workunit.client.0.smithi032.stdout:9 2024-09-13T11:57:33.773 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:57:33.773 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:57:34.210 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:57:35.211 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:57:35.211 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:57:35.212 INFO:tasks.workunit.client.0.smithi032.stdout:10 2024-09-13T11:57:35.212 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T11:57:35.212 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:57:35.212 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:57:35.624 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:57:36.624 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:57:36.625 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:57:36.625 INFO:tasks.workunit.client.0.smithi032.stdout:11 2024-09-13T11:57:36.625 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T11:57:36.626 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:57:36.626 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:57:37.224 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:57:38.226 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:57:38.323 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:57:38.324 INFO:tasks.workunit.client.0.smithi032.stdout:12 2024-09-13T11:57:38.324 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T11:57:38.324 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:57:38.325 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:57:38.979 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:57:39.981 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:57:39.981 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:57:39.981 INFO:tasks.workunit.client.0.smithi032.stdout:13 2024-09-13T11:57:39.981 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T11:57:39.982 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:57:39.982 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:57:40.750 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:57:41.752 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:57:41.752 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:57:41.752 INFO:tasks.workunit.client.0.smithi032.stdout:14 2024-09-13T11:57:41.753 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T11:57:41.753 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:57:41.753 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:57:42.464 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:57:43.466 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:57:43.466 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:57:43.466 INFO:tasks.workunit.client.0.smithi032.stdout:15 2024-09-13T11:57:43.467 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T11:57:43.467 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:57:43.467 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:57:44.030 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:57:45.032 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:57:45.032 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:57:45.032 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T11:57:45.032 INFO:tasks.workunit.client.0.smithi032.stdout:16 2024-09-13T11:57:45.033 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:57:45.033 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:57:46.226 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:57:47.227 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:57:47.228 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:57:47.228 INFO:tasks.workunit.client.0.smithi032.stdout:17 2024-09-13T11:57:47.228 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T11:57:47.228 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:57:47.229 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:57:47.688 INFO:tasks.workunit.client.0.smithi032.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2799830465,v1:127.0.0.1:6811/2799830465] [v2:127.0.0.1:6812/2799830465,v1:127.0.0.1:6813/2799830465] exists,up d577bf33-7894-4c3b-805e-9d8bc779f2a5 2024-09-13T11:57:47.692 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T11:57:47.692 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T11:57:47.692 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T11:57:47.692 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:32: TEST_bad_inc_map: run_osd td/bad-inc-map 2 2024-09-13T11:57:47.692 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/bad-inc-map 2024-09-13T11:57:47.692 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T11:57:47.692 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T11:57:47.692 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T11:57:47.693 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/bad-inc-map/2 2024-09-13T11:57:47.693 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e51e787f-f85e-4d21-9be8-7d13be4336e6 --auth-supported=none --mon-host=127.0.0.1:41173 ' 2024-09-13T11:57:47.693 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T11:57:47.693 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T11:57:47.693 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T11:57:47.693 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/2' 2024-09-13T11:57:47.693 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/2/journal' 2024-09-13T11:57:47.693 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T11:57:47.693 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T11:57:47.693 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-09-13T11:57:47.693 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T11:57:47.693 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T11:57:47.693 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T11:57:47.693 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T11:57:47.694 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T11:57:47.694 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27402 2024-09-13T11:57:47.694 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27402/$cluster-$name.asok' 2024-09-13T11:57:47.694 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27402/$cluster-$name.asok' 2024-09-13T11:57:47.694 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T11:57:47.694 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T11:57:47.694 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T11:57:47.694 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-09-13T11:57:47.694 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-09-13T11:57:47.694 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T11:57:47.694 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T11:57:47.694 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T11:57:47.694 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T11:57:47.694 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T11:57:47.695 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T11:57:47.695 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/bad-inc-map/2 2024-09-13T11:57:47.695 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T11:57:47.695 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0f3c04c7-4d35-4600-ade2-fbd7cfbb4ae8 2024-09-13T11:57:47.695 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 0f3c04c7-4d35-4600-ade2-fbd7cfbb4ae8' 2024-09-13T11:57:47.695 INFO:tasks.workunit.client.0.smithi032.stdout:add osd2 0f3c04c7-4d35-4600-ade2-fbd7cfbb4ae8 2024-09-13T11:57:47.695 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T11:57:47.710 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC7KORmJ1tHKhAANUCymRn8aSP1aLVRnaaGcg== 2024-09-13T11:57:47.710 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC7KORmJ1tHKhAANUCymRn8aSP1aLVRnaaGcg=="}' 2024-09-13T11:57:47.711 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0f3c04c7-4d35-4600-ade2-fbd7cfbb4ae8 -i td/bad-inc-map/2/new.json 2024-09-13T11:57:48.290 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T11:57:48.301 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/bad-inc-map/2/new.json 2024-09-13T11:57:48.302 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=e51e787f-f85e-4d21-9be8-7d13be4336e6 --auth-supported=none --mon-host=127.0.0.1:41173 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.27402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC7KORmJ1tHKhAANUCymRn8aSP1aLVRnaaGcg== --osd-uuid 0f3c04c7-4d35-4600-ade2-fbd7cfbb4ae8 2024-09-13T11:57:48.329 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:57:48.328+0000 7f1b34e0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:57:48.331 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:57:48.330+0000 7f1b34e0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:57:48.332 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:57:48.331+0000 7f1b34e0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:57:48.333 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:57:48.332+0000 7f1b34e0b640 -1 bdev(0x5585ed191c00 td/bad-inc-map/2/block) open stat got: (1) Operation not permitted 2024-09-13T11:57:48.333 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:57:48.332+0000 7f1b34e0b640 -1 bluestore(td/bad-inc-map/2) _read_fsid unparsable uuid 2024-09-13T11:57:54.633 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/bad-inc-map/2/keyring 2024-09-13T11:57:54.633 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T11:57:54.634 INFO:tasks.workunit.client.0.smithi032.stdout:adding osd2 key to auth repository 2024-09-13T11:57:54.634 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T11:57:54.634 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/bad-inc-map/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T11:57:55.194 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.2 2024-09-13T11:57:55.195 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T11:57:55.195 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=e51e787f-f85e-4d21-9be8-7d13be4336e6 --auth-supported=none --mon-host=127.0.0.1:41173 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.27402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T11:57:55.195 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T11:57:55.196 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T11:57:55.196 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T11:57:55.220 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:57:55.219+0000 7fe311640640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:57:55.247 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:57:55.246+0000 7fe311640640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:57:55.248 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:57:55.247+0000 7fe311640640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:57:55.669 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T11:57:55.669 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T11:57:55.669 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T11:57:55.669 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T11:57:55.669 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T11:57:55.669 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:57:55.670 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T11:57:55.670 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T11:57:55.670 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:57:55.670 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T11:57:56.137 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:57:56.362 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:57:56.361+0000 7fe311640640 -1 Falling back to public interface 2024-09-13T11:57:57.138 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:57:57.138 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:57:57.138 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T11:57:57.138 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T11:57:57.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:57:57.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T11:57:57.572 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:57:58.574 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:57:58.574 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:57:58.574 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T11:57:58.574 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T11:57:58.574 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:57:58.575 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T11:57:59.074 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:57:59.478 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:57:59.476+0000 7fe311640640 -1 osd.2 0 log_to_monitors true 2024-09-13T11:58:00.075 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:58:00.076 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:58:00.076 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T11:58:00.076 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T11:58:00.076 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:58:00.076 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T11:58:00.532 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:58:01.534 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:58:01.534 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:58:01.535 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T11:58:01.535 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T11:58:01.535 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:58:01.535 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T11:58:01.954 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:58:02.955 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:58:02.956 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:58:02.956 INFO:tasks.workunit.client.0.smithi032.stdout:5 2024-09-13T11:58:02.956 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T11:58:02.957 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:58:02.957 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T11:58:03.413 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:58:04.414 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:58:04.415 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:58:04.415 INFO:tasks.workunit.client.0.smithi032.stdout:6 2024-09-13T11:58:04.416 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T11:58:04.416 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:58:04.416 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T11:58:04.885 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:58:05.886 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:58:05.886 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:58:05.886 INFO:tasks.workunit.client.0.smithi032.stdout:7 2024-09-13T11:58:05.887 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T11:58:05.887 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:58:05.887 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T11:58:06.424 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:58:07.426 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:58:07.426 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:58:07.426 INFO:tasks.workunit.client.0.smithi032.stdout:8 2024-09-13T11:58:07.427 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T11:58:07.427 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:58:07.427 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T11:58:07.841 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:58:08.843 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:58:08.843 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:58:08.843 INFO:tasks.workunit.client.0.smithi032.stdout:9 2024-09-13T11:58:08.844 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T11:58:08.844 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:58:08.844 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T11:58:09.257 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:58:10.258 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:58:10.258 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:58:10.258 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T11:58:10.259 INFO:tasks.workunit.client.0.smithi032.stdout:10 2024-09-13T11:58:10.259 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:58:10.259 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T11:58:10.716 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:58:11.717 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:58:11.717 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:58:11.718 INFO:tasks.workunit.client.0.smithi032.stdout:11 2024-09-13T11:58:11.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T11:58:11.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:58:11.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T11:58:12.431 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:58:13.432 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:58:13.433 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:58:13.433 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T11:58:13.433 INFO:tasks.workunit.client.0.smithi032.stdout:12 2024-09-13T11:58:13.433 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:58:13.433 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T11:58:14.539 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:58:15.541 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:58:15.541 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:58:15.541 INFO:tasks.workunit.client.0.smithi032.stdout:13 2024-09-13T11:58:15.541 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T11:58:15.542 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:58:15.542 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T11:58:16.225 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:58:17.226 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:58:17.227 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:58:17.227 INFO:tasks.workunit.client.0.smithi032.stdout:14 2024-09-13T11:58:17.227 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T11:58:17.227 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:58:17.227 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T11:58:18.235 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:58:19.236 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:58:19.237 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:58:19.237 INFO:tasks.workunit.client.0.smithi032.stdout:15 2024-09-13T11:58:19.237 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T11:58:19.237 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:58:19.237 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T11:58:20.098 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:58:21.100 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:58:21.100 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:58:21.100 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T11:58:21.100 INFO:tasks.workunit.client.0.smithi032.stdout:16 2024-09-13T11:58:21.101 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:58:21.101 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T11:58:21.735 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:58:22.736 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:58:22.736 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:58:22.737 INFO:tasks.workunit.client.0.smithi032.stdout:17 2024-09-13T11:58:22.737 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T11:58:22.737 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:58:22.737 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T11:58:23.250 INFO:tasks.workunit.client.0.smithi032.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1952656485,v1:127.0.0.1:6819/1952656485] [v2:127.0.0.1:6820/1952656485,v1:127.0.0.1:6821/1952656485] exists,up 0f3c04c7-4d35-4600-ade2-fbd7cfbb4ae8 2024-09-13T11:58:23.250 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T11:58:23.250 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T11:58:23.251 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T11:58:23.251 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:34: TEST_bad_inc_map: ceph config set osd.2 osd_inject_bad_map_crc_probability 1 2024-09-13T11:58:23.717 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:37: TEST_bad_inc_map: create_pool foo 8 2024-09-13T11:58:23.717 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 8 2024-09-13T11:58:24.271 INFO:tasks.workunit.client.0.smithi032.stderr:pool 'foo' already exists 2024-09-13T11:58:24.282 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T11:58:25.283 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:38: TEST_bad_inc_map: ceph osd pool set foo min_size 1 2024-09-13T11:58:26.707 INFO:tasks.workunit.client.0.smithi032.stderr:set pool 1 min_size to 1 2024-09-13T11:58:26.722 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:39: TEST_bad_inc_map: ceph osd pool set foo min_size 2 2024-09-13T11:58:28.041 INFO:tasks.workunit.client.0.smithi032.stderr:set pool 1 min_size to 2 2024-09-13T11:58:28.054 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:41: TEST_bad_inc_map: sleep 5 2024-09-13T11:58:33.055 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: TIMEOUT=10 2024-09-13T11:58:33.056 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: wait_for_osd up 0 2024-09-13T11:58:33.056 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T11:58:33.056 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T11:58:33.056 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T11:58:33.056 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T11:58:33.056 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 10 )) 2024-09-13T11:58:33.056 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T11:58:33.057 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T11:58:33.057 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:58:33.057 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:58:33.500 INFO:tasks.workunit.client.0.smithi032.stdout:osd.0 up in weight 1 up_from 5 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2540798877,v1:127.0.0.1:6803/2540798877] [v2:127.0.0.1:6804/2540798877,v1:127.0.0.1:6805/2540798877] exists,up 0651a177-a4aa-4fba-8e0b-e3c6498b30e7 2024-09-13T11:58:33.501 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T11:58:33.501 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T11:58:33.501 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T11:58:33.502 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: TIMEOUT=10 2024-09-13T11:58:33.502 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: wait_for_osd up 1 2024-09-13T11:58:33.502 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T11:58:33.502 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T11:58:33.502 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T11:58:33.502 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T11:58:33.502 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 10 )) 2024-09-13T11:58:33.502 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T11:58:33.503 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T11:58:33.503 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:58:33.503 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:58:33.935 INFO:tasks.workunit.client.0.smithi032.stdout:osd.1 up in weight 1 up_from 10 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2799830465,v1:127.0.0.1:6811/2799830465] [v2:127.0.0.1:6812/2799830465,v1:127.0.0.1:6813/2799830465] exists,up d577bf33-7894-4c3b-805e-9d8bc779f2a5 2024-09-13T11:58:33.937 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T11:58:33.937 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T11:58:33.937 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T11:58:33.937 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: TIMEOUT=10 2024-09-13T11:58:33.937 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: wait_for_osd up 2 2024-09-13T11:58:33.937 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T11:58:33.937 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T11:58:33.937 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T11:58:33.937 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T11:58:33.938 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 10 )) 2024-09-13T11:58:33.938 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T11:58:33.938 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T11:58:33.938 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:58:33.938 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T11:58:34.361 INFO:tasks.workunit.client.0.smithi032.stdout:osd.2 up in weight 1 up_from 15 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1952656485,v1:127.0.0.1:6819/1952656485] [v2:127.0.0.1:6820/1952656485,v1:127.0.0.1:6821/1952656485] exists,up 0f3c04c7-4d35-4600-ade2-fbd7cfbb4ae8 2024-09-13T11:58:34.362 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T11:58:34.362 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T11:58:34.362 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T11:58:34.362 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:49: TEST_bad_inc_map: grep 'injecting map crc failure' td/bad-inc-map/osd.2.log 2024-09-13T11:58:34.364 INFO:tasks.workunit.client.0.smithi032.stdout:2024-09-13T11:58:24.120+0000 7fe3025cc640 -1 osd.2 15 handle_osd_map injecting map crc failure 2024-09-13T11:58:34.364 INFO:tasks.workunit.client.0.smithi032.stdout:2024-09-13T11:58:24.374+0000 7fe3025cc640 -1 osd.2 17 handle_osd_map injecting map crc failure 2024-09-13T11:58:34.364 INFO:tasks.workunit.client.0.smithi032.stdout:2024-09-13T11:58:25.144+0000 7fe3025cc640 -1 osd.2 18 handle_osd_map injecting map crc failure 2024-09-13T11:58:34.364 INFO:tasks.workunit.client.0.smithi032.stdout:2024-09-13T11:58:26.143+0000 7fe3005c8640 -1 osd.2 19 handle_osd_map injecting map crc failure 2024-09-13T11:58:34.364 INFO:tasks.workunit.client.0.smithi032.stdout:2024-09-13T11:58:26.546+0000 7fe3005c8640 -1 osd.2 22 handle_osd_map injecting map crc failure 2024-09-13T11:58:34.364 INFO:tasks.workunit.client.0.smithi032.stdout:2024-09-13T11:58:26.730+0000 7fe3025cc640 -1 osd.2 23 handle_osd_map injecting map crc failure 2024-09-13T11:58:34.364 INFO:tasks.workunit.client.0.smithi032.stdout:2024-09-13T11:58:27.494+0000 7fe3025cc640 -1 osd.2 24 handle_osd_map injecting map crc failure 2024-09-13T11:58:34.364 INFO:tasks.workunit.client.0.smithi032.stdout:2024-09-13T11:58:28.064+0000 7fe3025cc640 -1 osd.2 25 handle_osd_map injecting map crc failure 2024-09-13T11:58:34.364 INFO:tasks.workunit.client.0.smithi032.stdout:2024-09-13T11:58:28.066+0000 7fe3025cc640 -1 osd.2 25 handle_osd_map injecting map crc failure 2024-09-13T11:58:34.364 INFO:tasks.workunit.client.0.smithi032.stdout:2024-09-13T11:58:29.402+0000 7fe3005c8640 -1 osd.2 26 handle_osd_map injecting map crc failure 2024-09-13T11:58:34.365 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:50: TEST_bad_inc_map: grep 'bailing because last' td/bad-inc-map/osd.2.log 2024-09-13T11:58:34.366 INFO:tasks.workunit.client.0.smithi032.stdout:2024-09-13T11:58:24.121+0000 7fe3025cc640 10 osd.2 15 handle_osd_map bailing because last < start (15<16) 2024-09-13T11:58:34.366 INFO:tasks.workunit.client.0.smithi032.stdout:2024-09-13T11:58:24.374+0000 7fe3025cc640 10 osd.2 17 handle_osd_map bailing because last < start (17<18) 2024-09-13T11:58:34.366 INFO:tasks.workunit.client.0.smithi032.stdout:2024-09-13T11:58:25.144+0000 7fe3025cc640 10 osd.2 18 handle_osd_map bailing because last < start (18<19) 2024-09-13T11:58:34.366 INFO:tasks.workunit.client.0.smithi032.stdout:2024-09-13T11:58:26.144+0000 7fe3005c8640 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2024-09-13T11:58:34.366 INFO:tasks.workunit.client.0.smithi032.stdout:2024-09-13T11:58:26.546+0000 7fe3005c8640 10 osd.2 22 handle_osd_map bailing because last < start (22<23) 2024-09-13T11:58:34.366 INFO:tasks.workunit.client.0.smithi032.stdout:2024-09-13T11:58:26.730+0000 7fe3025cc640 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2024-09-13T11:58:34.366 INFO:tasks.workunit.client.0.smithi032.stdout:2024-09-13T11:58:27.494+0000 7fe3025cc640 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2024-09-13T11:58:34.366 INFO:tasks.workunit.client.0.smithi032.stdout:2024-09-13T11:58:28.065+0000 7fe3025cc640 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2024-09-13T11:58:34.367 INFO:tasks.workunit.client.0.smithi032.stdout:2024-09-13T11:58:28.066+0000 7fe3025cc640 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2024-09-13T11:58:34.367 INFO:tasks.workunit.client.0.smithi032.stdout:2024-09-13T11:58:29.402+0000 7fe3005c8640 10 osd.2 26 handle_osd_map bailing because last < start (26<27) 2024-09-13T11:58:34.367 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:52: TEST_bad_inc_map: echo success 2024-09-13T11:58:34.367 INFO:tasks.workunit.client.0.smithi032.stdout:success 2024-09-13T11:58:34.368 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:54: TEST_bad_inc_map: delete_pool foo 2024-09-13T11:58:34.368 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=foo 2024-09-13T11:58:34.368 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-09-13T11:58:34.895 INFO:tasks.workunit.client.0.smithi032.stderr:pool 'foo' does not exist 2024-09-13T11:58:34.907 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:55: TEST_bad_inc_map: kill_daemons td/bad-inc-map 2024-09-13T11:58:34.908 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T11:58:34.908 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T11:58:34.908 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T11:58:34.908 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T11:58:34.908 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T11:58:40.221 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T11:58:40.222 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:21: run: teardown td/bad-inc-map 2024-09-13T11:58:40.222 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/bad-inc-map 2024-09-13T11:58:40.222 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T11:58:40.222 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/bad-inc-map KILL 2024-09-13T11:58:40.222 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T11:58:40.222 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T11:58:40.222 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T11:58:40.222 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T11:58:40.222 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T11:58:40.227 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T11:58:40.228 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T11:58:40.229 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T11:58:40.229 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T11:58:40.230 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T11:58:40.230 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T11:58:40.230 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T11:58:40.232 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:58:40.232 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T11:58:40.232 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T11:58:40.232 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:58:40.233 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T11:58:40.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T11:58:40.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/bad-inc-map 2024-09-13T11:58:40.250 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T11:58:40.250 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T11:58:40.250 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27402 2024-09-13T11:58:40.250 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27402 2024-09-13T11:58:40.251 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T11:58:40.252 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T11:58:40.252 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-13T11:58:40.252 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/bad-inc-map 0 2024-09-13T11:58:40.252 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/bad-inc-map 2024-09-13T11:58:40.252 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-13T11:58:40.252 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/bad-inc-map KILL 2024-09-13T11:58:40.252 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T11:58:40.252 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T11:58:40.252 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T11:58:40.252 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T11:58:40.252 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T11:58:40.253 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T11:58:40.254 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T11:58:40.254 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T11:58:40.255 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T11:58:40.256 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T11:58:40.256 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T11:58:40.256 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T11:58:40.257 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:58:40.257 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T11:58:40.258 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T11:58:40.258 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:58:40.258 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T11:58:40.260 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-13T11:58:40.260 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/bad-inc-map 2024-09-13T11:58:40.261 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T11:58:40.261 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T11:58:40.261 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27402 2024-09-13T11:58:40.261 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27402 2024-09-13T11:58:40.263 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T11:58:40.264 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T11:58:40.264 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-13T11:58:40.264 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-13T11:58:40.264 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-13T11:58:40.293 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2024-09-13T11:58:40.293 DEBUG:teuthology.orchestra.run.smithi032:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d58f73069a8157a9ace12a0247ee3e8497707ea9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2024-09-13T11:58:40.357 INFO:tasks.workunit.client.0.smithi032.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-13T11:58:40.364 INFO:tasks.workunit.client.0.smithi032.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-13T11:58:40.364 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-13T11:58:40.364 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-13T11:58:40.364 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-13T11:58:40.364 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-13T11:58:40.364 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-13T11:58:40.364 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-13T11:58:40.364 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-13T11:58:40.364 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-13T11:58:40.364 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/divergent-priors 2024-09-13T11:58:40.364 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2024-09-13T11:58:40.365 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2024-09-13T11:58:40.365 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2024-09-13T11:58:40.365 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2024-09-13T11:58:40.365 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2024-09-13T11:58:40.365 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2024-09-13T11:58:40.365 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2024-09-13T11:58:40.366 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2024-09-13T11:58:40.366 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2024-09-13T11:58:40.366 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2024-09-13T11:58:40.366 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2024-09-13T11:58:40.366 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2024-09-13T11:58:40.366 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2024-09-13T11:58:40.367 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2024-09-13T11:58:40.367 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2024-09-13T11:58:40.367 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2024-09-13T11:58:40.368 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2024-09-13T11:58:40.368 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2024-09-13T11:58:40.369 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2024-09-13T11:58:40.369 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2024-09-13T11:58:40.369 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2024-09-13T11:58:40.369 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2024-09-13T11:58:40.369 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2024-09-13T11:58:40.369 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2024-09-13T11:58:40.369 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2024-09-13T11:58:40.369 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2024-09-13T11:58:40.370 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none ' 2024-09-13T11:58:40.370 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-09-13T11:58:40.370 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-09-13T11:58:40.370 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2024-09-13T11:58:40.370 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T11:58:40.370 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2024-09-13T11:58:40.370 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-13T11:58:40.373 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2024-09-13T11:58:40.373 INFO:tasks.workunit.client.0.smithi032.stderr:TEST_divergent_2 2024-09-13T11:58:40.373 INFO:tasks.workunit.client.0.smithi032.stderr:TEST_divergent_3 2024-09-13T11:58:40.373 INFO:tasks.workunit.client.0.smithi032.stderr:TEST_divergent_ec' 2024-09-13T11:58:40.373 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-13T11:58:40.373 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-13T11:58:40.373 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/divergent-priors 2024-09-13T11:58:40.373 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/divergent-priors 2024-09-13T11:58:40.373 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-13T11:58:40.373 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T11:58:40.374 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-13T11:58:40.374 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T11:58:40.374 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T11:58:40.374 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T11:58:40.374 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T11:58:40.374 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T11:58:40.375 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T11:58:40.375 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T11:58:40.376 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T11:58:40.376 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T11:58:40.376 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T11:58:40.376 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T11:58:40.377 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T11:58:40.378 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:58:40.378 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T11:58:40.378 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T11:58:40.378 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:58:40.378 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T11:58:40.380 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T11:58:40.380 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-13T11:58:40.381 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T11:58:40.381 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T11:58:40.381 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T11:58:40.381 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.33242 2024-09-13T11:58:40.382 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T11:58:40.382 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T11:58:40.382 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/divergent-priors 2024-09-13T11:58:40.383 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T11:58:40.383 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T11:58:40.384 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T11:58:40.384 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.33242 2024-09-13T11:58:40.385 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T11:58:40.386 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-13T11:58:40.386 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-13T11:58:40.386 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T11:58:40.386 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-13T11:58:40.386 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2024-09-13T11:58:40.386 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2024-09-13T11:58:40.386 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2024-09-13T11:58:40.386 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2024-09-13T11:58:40.386 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2024-09-13T11:58:40.386 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2024-09-13T11:58:40.387 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2024-09-13T11:58:40.396 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2024-09-13T11:58:40.397 INFO:tasks.workunit.client.0.smithi032.stderr:1 2024-09-13T11:58:40.397 INFO:tasks.workunit.client.0.smithi032.stderr:2' 2024-09-13T11:58:40.397 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2024-09-13T11:58:40.397 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/divergent-priors 2024-09-13T11:58:40.397 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T11:58:40.397 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T11:58:40.397 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T11:58:40.397 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/divergent-priors/a 2024-09-13T11:58:40.397 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-13T11:58:40.844 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T11:58:40.844 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T11:58:40.844 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T11:58:40.844 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T11:58:40.844 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T11:58:40.845 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T11:58:40.845 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T11:58:40.845 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-13T11:58:41.055 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T11:58:41.056 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T11:58:41.056 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T11:58:41.056 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T11:58:41.056 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T11:58:41.057 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T11:58:41.057 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T11:58:41.057 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T11:58:41.057 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T11:58:41.058 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T11:58:41.058 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T11:58:41.058 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T11:58:41.058 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-mon.a.asok 2024-09-13T11:58:41.058 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T11:58:41.058 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.33242/ceph-mon.a.asok config get fsid 2024-09-13T11:58:41.141 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T11:58:41.141 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T11:58:41.141 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T11:58:41.141 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T11:58:41.141 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T11:58:41.141 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T11:58:41.141 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T11:58:41.141 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T11:58:41.141 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T11:58:41.141 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T11:58:41.142 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T11:58:41.142 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-mon.a.asok 2024-09-13T11:58:41.142 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T11:58:41.142 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.33242/ceph-mon.a.asok config get mon_host 2024-09-13T11:58:41.223 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2024-09-13T11:58:41.223 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/divergent-priors 2024-09-13T11:58:41.223 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T11:58:41.223 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T11:58:41.223 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T11:58:41.223 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/divergent-priors/x 2024-09-13T11:58:41.223 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T11:58:42.103 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T11:58:42.103 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T11:58:42.104 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T11:58:42.104 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T11:58:42.104 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T11:58:42.104 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T11:58:42.104 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T11:58:42.104 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T11:58:42.104 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T11:58:42.141 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-09-13T11:58:42.141 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2024-09-13T11:58:42.141 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-13T11:58:42.141 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T11:58:42.141 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T11:58:42.141 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T11:58:42.141 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/0 2024-09-13T11:58:42.141 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T11:58:42.141 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T11:58:42.141 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T11:58:42.142 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T11:58:42.142 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-13T11:58:42.142 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-13T11:58:42.142 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T11:58:42.142 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T11:58:42.142 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T11:58:42.142 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T11:58:42.142 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T11:58:42.142 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T11:58:42.142 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T11:58:42.142 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T11:58:42.142 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T11:58:42.143 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T11:58:42.143 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T11:58:42.143 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T11:58:42.143 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T11:58:42.143 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T11:58:42.143 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T11:58:42.143 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T11:58:42.143 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T11:58:42.143 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T11:58:42.143 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T11:58:42.143 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T11:58:42.143 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T11:58:42.143 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T11:58:42.144 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/0 2024-09-13T11:58:42.144 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T11:58:42.144 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=42e90f08-426a-4b5b-ad78-7bd0b8144f62 2024-09-13T11:58:42.144 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 42e90f08-426a-4b5b-ad78-7bd0b8144f62' 2024-09-13T11:58:42.145 INFO:tasks.workunit.client.0.smithi032.stdout:add osd0 42e90f08-426a-4b5b-ad78-7bd0b8144f62 2024-09-13T11:58:42.145 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T11:58:42.166 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDyKORm6VjYCRAAw9Rz4op1GP3NT8kI9lrXMA== 2024-09-13T11:58:42.167 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDyKORm6VjYCRAAw9Rz4op1GP3NT8kI9lrXMA=="}' 2024-09-13T11:58:42.167 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 42e90f08-426a-4b5b-ad78-7bd0b8144f62 -i td/divergent-priors/0/new.json 2024-09-13T11:58:42.429 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T11:58:42.439 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/0/new.json 2024-09-13T11:58:42.439 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDyKORm6VjYCRAAw9Rz4op1GP3NT8kI9lrXMA== --osd-uuid 42e90f08-426a-4b5b-ad78-7bd0b8144f62 2024-09-13T11:58:42.477 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:58:42.476+0000 7f3b6a883640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:58:42.497 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:58:42.497+0000 7f3b6a883640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:58:42.499 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:58:42.498+0000 7f3b6a883640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:58:42.499 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:58:42.498+0000 7f3b6a883640 -1 bdev(0x561ba02bfc00 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-09-13T11:58:42.499 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:58:42.498+0000 7f3b6a883640 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-13T11:58:48.205 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-13T11:58:48.205 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T11:58:48.206 INFO:tasks.workunit.client.0.smithi032.stdout:adding osd0 key to auth repository 2024-09-13T11:58:48.206 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T11:58:48.206 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T11:58:48.778 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.0 2024-09-13T11:58:48.779 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T11:58:48.779 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T11:58:48.779 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T11:58:48.779 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T11:58:48.779 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T11:58:48.814 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:58:48.813+0000 7fdf4ce0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:58:48.816 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:58:48.815+0000 7fdf4ce0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:58:48.817 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:58:48.816+0000 7fdf4ce0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:58:49.182 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T11:58:49.182 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T11:58:49.183 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T11:58:49.183 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T11:58:49.183 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T11:58:49.183 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:58:49.183 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T11:58:49.183 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T11:58:49.183 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:58:49.184 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:58:49.644 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:58:49.673 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:58:49.672+0000 7fdf4ce0b640 -1 Falling back to public interface 2024-09-13T11:58:50.645 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:58:50.646 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:58:50.646 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T11:58:50.646 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T11:58:50.646 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:58:50.647 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:58:51.095 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:58:52.097 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:58:52.097 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:58:52.097 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T11:58:52.098 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T11:58:52.098 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:58:52.098 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:58:52.515 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:58:52.641 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:58:52.640+0000 7fdf4ce0b640 -1 osd.0 0 log_to_monitors true 2024-09-13T11:58:53.516 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:58:53.516 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:58:53.516 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T11:58:53.517 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T11:58:53.517 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:58:53.517 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:58:53.994 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:58:54.995 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:58:54.996 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:58:54.996 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T11:58:54.996 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T11:58:54.997 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:58:54.997 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:58:55.452 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:58:56.454 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:58:56.454 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:58:56.454 INFO:tasks.workunit.client.0.smithi032.stdout:5 2024-09-13T11:58:56.455 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T11:58:56.455 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:58:56.455 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:58:56.893 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:58:57.894 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:58:57.894 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:58:57.894 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T11:58:57.894 INFO:tasks.workunit.client.0.smithi032.stdout:6 2024-09-13T11:58:57.895 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:58:57.895 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:58:58.460 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:58:59.462 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:58:59.462 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:58:59.462 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T11:58:59.462 INFO:tasks.workunit.client.0.smithi032.stdout:7 2024-09-13T11:58:59.463 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:58:59.463 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:58:59.911 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:59:00.912 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:59:00.912 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:59:00.912 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T11:59:00.913 INFO:tasks.workunit.client.0.smithi032.stdout:8 2024-09-13T11:59:00.913 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:59:00.913 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:59:01.437 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:59:02.438 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:59:02.438 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:59:02.439 INFO:tasks.workunit.client.0.smithi032.stdout:9 2024-09-13T11:59:02.439 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T11:59:02.439 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:59:02.439 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:59:02.856 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:59:03.857 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:59:03.857 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:59:03.858 INFO:tasks.workunit.client.0.smithi032.stdout:10 2024-09-13T11:59:03.858 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T11:59:03.858 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:59:03.858 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:59:04.472 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:59:05.473 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:59:05.473 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:59:05.474 INFO:tasks.workunit.client.0.smithi032.stdout:11 2024-09-13T11:59:05.474 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T11:59:05.474 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:59:05.475 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:59:05.966 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:59:06.968 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:59:06.968 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:59:06.968 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T11:59:06.968 INFO:tasks.workunit.client.0.smithi032.stdout:12 2024-09-13T11:59:06.969 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:59:06.969 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:59:07.649 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:59:08.647 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:59:08.648 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:59:08.648 INFO:tasks.workunit.client.0.smithi032.stdout:13 2024-09-13T11:59:08.648 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T11:59:08.649 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:59:08.649 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:59:09.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:59:10.736 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:59:10.736 INFO:tasks.workunit.client.0.smithi032.stdout:14 2024-09-13T11:59:10.737 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:59:10.737 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T11:59:10.737 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:59:10.737 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:59:11.580 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:59:12.582 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:59:12.582 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:59:12.582 INFO:tasks.workunit.client.0.smithi032.stdout:15 2024-09-13T11:59:12.583 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T11:59:12.583 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:59:12.583 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:59:13.882 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:59:14.883 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:59:14.883 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:59:14.883 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T11:59:14.883 INFO:tasks.workunit.client.0.smithi032.stdout:16 2024-09-13T11:59:14.884 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:59:14.884 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T11:59:15.378 INFO:tasks.workunit.client.0.smithi032.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3540069113,v1:127.0.0.1:6803/3540069113] [v2:127.0.0.1:6804/3540069113,v1:127.0.0.1:6805/3540069113] exists,up 42e90f08-426a-4b5b-ad78-7bd0b8144f62 2024-09-13T11:59:15.382 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T11:59:15.382 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T11:59:15.382 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T11:59:15.382 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-09-13T11:59:15.382 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2024-09-13T11:59:15.382 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-13T11:59:15.383 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T11:59:15.383 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T11:59:15.383 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T11:59:15.383 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/1 2024-09-13T11:59:15.383 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T11:59:15.383 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T11:59:15.383 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T11:59:15.383 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T11:59:15.383 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-13T11:59:15.383 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-13T11:59:15.383 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T11:59:15.384 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T11:59:15.384 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T11:59:15.384 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T11:59:15.384 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T11:59:15.384 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T11:59:15.384 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T11:59:15.384 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T11:59:15.384 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T11:59:15.384 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T11:59:15.384 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T11:59:15.384 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T11:59:15.384 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T11:59:15.385 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T11:59:15.385 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T11:59:15.385 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T11:59:15.385 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T11:59:15.385 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T11:59:15.385 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T11:59:15.385 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T11:59:15.385 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T11:59:15.385 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T11:59:15.385 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/1 2024-09-13T11:59:15.385 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T11:59:15.385 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=45eb455b-ec14-492a-a47e-4e0a84ac4a8c 2024-09-13T11:59:15.386 INFO:tasks.workunit.client.0.smithi032.stdout:add osd1 45eb455b-ec14-492a-a47e-4e0a84ac4a8c 2024-09-13T11:59:15.386 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 45eb455b-ec14-492a-a47e-4e0a84ac4a8c' 2024-09-13T11:59:15.386 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T11:59:15.406 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQATKeRmcIErGBAA8lHES4kMntiXE3Baxiy1zQ== 2024-09-13T11:59:15.407 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQATKeRmcIErGBAA8lHES4kMntiXE3Baxiy1zQ=="}' 2024-09-13T11:59:15.407 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 45eb455b-ec14-492a-a47e-4e0a84ac4a8c -i td/divergent-priors/1/new.json 2024-09-13T11:59:15.942 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T11:59:15.954 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/1/new.json 2024-09-13T11:59:15.955 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQATKeRmcIErGBAA8lHES4kMntiXE3Baxiy1zQ== --osd-uuid 45eb455b-ec14-492a-a47e-4e0a84ac4a8c 2024-09-13T11:59:15.992 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:59:15.990+0000 7f71e254f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:59:15.995 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:59:15.993+0000 7f71e254f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:59:15.996 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:59:15.994+0000 7f71e254f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:59:15.996 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:59:15.994+0000 7f71e254f640 -1 bdev(0x5588bc0ffc00 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-09-13T11:59:15.996 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:59:15.995+0000 7f71e254f640 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-13T11:59:21.952 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-13T11:59:21.952 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T11:59:21.953 INFO:tasks.workunit.client.0.smithi032.stdout:adding osd1 key to auth repository 2024-09-13T11:59:21.954 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T11:59:21.954 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T11:59:22.539 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.1 2024-09-13T11:59:22.540 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T11:59:22.540 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T11:59:22.540 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T11:59:22.540 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T11:59:22.540 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T11:59:22.575 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:59:22.574+0000 7f6a3ca60640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:59:22.577 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:59:22.576+0000 7f6a3ca60640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:59:22.578 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:59:22.577+0000 7f6a3ca60640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:59:23.005 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T11:59:23.005 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T11:59:23.005 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T11:59:23.005 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T11:59:23.006 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T11:59:23.006 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:59:23.006 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T11:59:23.006 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T11:59:23.006 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:59:23.006 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:59:23.413 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:59:23.412+0000 7f6a3ca60640 -1 Falling back to public interface 2024-09-13T11:59:23.473 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:59:24.475 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:59:24.475 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:59:24.475 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T11:59:24.475 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T11:59:24.475 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:59:24.476 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:59:24.925 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:59:25.813 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:59:25.812+0000 7f6a3ca60640 -1 osd.1 0 log_to_monitors true 2024-09-13T11:59:25.926 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:59:25.927 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:59:25.927 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T11:59:25.927 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T11:59:25.927 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:59:25.928 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:59:26.565 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:59:27.567 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:59:27.567 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:59:27.567 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T11:59:27.567 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T11:59:27.568 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:59:27.568 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:59:28.123 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:59:29.124 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:59:29.124 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:59:29.124 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T11:59:29.125 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T11:59:29.125 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:59:29.125 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:59:29.590 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:59:30.591 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:59:30.592 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:59:30.592 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T11:59:30.592 INFO:tasks.workunit.client.0.smithi032.stdout:5 2024-09-13T11:59:30.593 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:59:30.593 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:59:31.021 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:59:32.022 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:59:32.022 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:59:32.023 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T11:59:32.023 INFO:tasks.workunit.client.0.smithi032.stdout:6 2024-09-13T11:59:32.023 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:59:32.023 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:59:32.540 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:59:33.542 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:59:33.542 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:59:33.542 INFO:tasks.workunit.client.0.smithi032.stdout:7 2024-09-13T11:59:33.543 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T11:59:33.543 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:59:33.543 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:59:33.960 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:59:34.961 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:59:34.961 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:59:34.961 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T11:59:34.962 INFO:tasks.workunit.client.0.smithi032.stdout:8 2024-09-13T11:59:34.962 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:59:34.962 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:59:35.456 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:59:36.457 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:59:36.458 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:59:36.458 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T11:59:36.458 INFO:tasks.workunit.client.0.smithi032.stdout:9 2024-09-13T11:59:36.459 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:59:36.459 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:59:36.944 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:59:37.945 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:59:37.945 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:59:37.945 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T11:59:37.946 INFO:tasks.workunit.client.0.smithi032.stdout:10 2024-09-13T11:59:37.946 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:59:37.946 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:59:38.541 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:59:39.542 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:59:39.543 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:59:39.543 INFO:tasks.workunit.client.0.smithi032.stdout:11 2024-09-13T11:59:39.543 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T11:59:39.543 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:59:39.543 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:59:40.283 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:59:41.284 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:59:41.285 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:59:41.285 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T11:59:41.285 INFO:tasks.workunit.client.0.smithi032.stdout:12 2024-09-13T11:59:41.285 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:59:41.285 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:59:41.974 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:59:42.976 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:59:42.976 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:59:42.976 INFO:tasks.workunit.client.0.smithi032.stdout:13 2024-09-13T11:59:42.977 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T11:59:42.977 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:59:42.977 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:59:44.002 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:59:45.003 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:59:45.003 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:59:45.003 INFO:tasks.workunit.client.0.smithi032.stdout:14 2024-09-13T11:59:45.004 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T11:59:45.004 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:59:45.004 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:59:45.765 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:59:46.766 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:59:46.766 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:59:46.766 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T11:59:46.766 INFO:tasks.workunit.client.0.smithi032.stdout:15 2024-09-13T11:59:46.767 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:59:46.767 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:59:47.598 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:59:48.599 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:59:48.599 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:59:48.599 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T11:59:48.600 INFO:tasks.workunit.client.0.smithi032.stdout:16 2024-09-13T11:59:48.600 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:59:48.600 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T11:59:49.081 INFO:tasks.workunit.client.0.smithi032.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/671786448,v1:127.0.0.1:6811/671786448] [v2:127.0.0.1:6812/671786448,v1:127.0.0.1:6813/671786448] exists,up 45eb455b-ec14-492a-a47e-4e0a84ac4a8c 2024-09-13T11:59:49.084 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T11:59:49.085 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T11:59:49.085 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T11:59:49.085 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-09-13T11:59:49.085 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2024-09-13T11:59:49.085 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-13T11:59:49.085 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T11:59:49.085 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T11:59:49.085 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T11:59:49.085 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/2 2024-09-13T11:59:49.085 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T11:59:49.085 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T11:59:49.085 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T11:59:49.085 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T11:59:49.086 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-13T11:59:49.086 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-13T11:59:49.086 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T11:59:49.086 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T11:59:49.086 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T11:59:49.086 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T11:59:49.086 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T11:59:49.086 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T11:59:49.086 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T11:59:49.086 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T11:59:49.086 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T11:59:49.086 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T11:59:49.086 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T11:59:49.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T11:59:49.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T11:59:49.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T11:59:49.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T11:59:49.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T11:59:49.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T11:59:49.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T11:59:49.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T11:59:49.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T11:59:49.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T11:59:49.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T11:59:49.088 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/2 2024-09-13T11:59:49.088 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T11:59:49.088 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4607d722-ea41-4ddb-a0cd-5570afca8326 2024-09-13T11:59:49.088 INFO:tasks.workunit.client.0.smithi032.stdout:add osd2 4607d722-ea41-4ddb-a0cd-5570afca8326 2024-09-13T11:59:49.088 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 4607d722-ea41-4ddb-a0cd-5570afca8326' 2024-09-13T11:59:49.088 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T11:59:49.109 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA1KeRm405zBhAAfz8GRcol9YsbRbbLlUdiKA== 2024-09-13T11:59:49.109 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA1KeRm405zBhAAfz8GRcol9YsbRbbLlUdiKA=="}' 2024-09-13T11:59:49.109 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4607d722-ea41-4ddb-a0cd-5570afca8326 -i td/divergent-priors/2/new.json 2024-09-13T11:59:49.574 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T11:59:49.586 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/2/new.json 2024-09-13T11:59:49.586 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA1KeRm405zBhAAfz8GRcol9YsbRbbLlUdiKA== --osd-uuid 4607d722-ea41-4ddb-a0cd-5570afca8326 2024-09-13T11:59:49.624 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:59:49.623+0000 7fb6ebd1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:59:49.626 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:59:49.625+0000 7fb6ebd1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:59:49.627 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:59:49.626+0000 7fb6ebd1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:59:49.628 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:59:49.627+0000 7fb6ebd1b640 -1 bdev(0x55d11c265c00 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-09-13T11:59:49.628 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:59:49.627+0000 7fb6ebd1b640 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-13T11:59:55.840 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-13T11:59:55.840 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T11:59:55.841 INFO:tasks.workunit.client.0.smithi032.stdout:adding osd2 key to auth repository 2024-09-13T11:59:55.841 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T11:59:55.841 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T11:59:56.419 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.2 2024-09-13T11:59:56.420 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T11:59:56.420 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T11:59:56.420 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T11:59:56.420 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T11:59:56.420 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T11:59:56.455 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:59:56.453+0000 7f1e81154640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:59:56.457 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:59:56.455+0000 7f1e81154640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:59:56.458 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:59:56.456+0000 7f1e81154640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T11:59:56.857 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T11:59:56.857 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T11:59:56.858 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T11:59:56.858 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T11:59:56.858 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T11:59:56.858 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:59:56.858 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T11:59:56.858 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T11:59:56.859 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:59:56.859 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T11:59:57.300 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:59:57.805 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T11:59:57.803+0000 7f1e81154640 -1 Falling back to public interface 2024-09-13T11:59:58.301 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:59:58.301 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:59:58.302 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T11:59:58.302 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T11:59:58.302 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:59:58.303 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T11:59:58.751 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T11:59:59.753 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T11:59:59.753 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T11:59:59.753 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T11:59:59.754 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T11:59:59.754 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T11:59:59.754 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:00:00.203 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:00:00.833 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:00:00.831+0000 7f1e81154640 -1 osd.2 0 log_to_monitors true 2024-09-13T12:00:01.204 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:00:01.204 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:00:01.204 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:00:01.205 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:00:01.205 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:00:01.205 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:00:01.643 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:00:02.644 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:00:02.645 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:00:02.645 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:00:02.645 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:00:02.645 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:00:02.645 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:00:03.092 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:00:04.093 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:00:04.093 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:00:04.093 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T12:00:04.094 INFO:tasks.workunit.client.0.smithi032.stdout:5 2024-09-13T12:00:04.094 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:00:04.094 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:00:04.531 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:00:05.532 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:00:05.532 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:00:05.533 INFO:tasks.workunit.client.0.smithi032.stdout:6 2024-09-13T12:00:05.533 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T12:00:05.533 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:00:05.533 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:00:06.029 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:00:07.030 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:00:07.031 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:00:07.031 INFO:tasks.workunit.client.0.smithi032.stdout:7 2024-09-13T12:00:07.031 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T12:00:07.032 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:00:07.032 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:00:07.470 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:00:08.471 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:00:08.472 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:00:08.472 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T12:00:08.472 INFO:tasks.workunit.client.0.smithi032.stdout:8 2024-09-13T12:00:08.473 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:00:08.473 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:00:08.893 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:00:09.895 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:00:09.895 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:00:09.895 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T12:00:09.895 INFO:tasks.workunit.client.0.smithi032.stdout:9 2024-09-13T12:00:09.895 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:00:09.896 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:00:10.322 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:00:11.323 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:00:11.324 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:00:11.324 INFO:tasks.workunit.client.0.smithi032.stdout:10 2024-09-13T12:00:11.324 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T12:00:11.325 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:00:11.325 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:00:11.891 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:00:12.893 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:00:12.893 INFO:tasks.workunit.client.0.smithi032.stdout:11 2024-09-13T12:00:12.893 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:00:12.894 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T12:00:12.894 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:00:12.894 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:00:13.343 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:00:14.345 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:00:14.345 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:00:14.345 INFO:tasks.workunit.client.0.smithi032.stdout:12 2024-09-13T12:00:14.346 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T12:00:14.346 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:00:14.346 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:00:14.916 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:00:15.918 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:00:15.918 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:00:15.918 INFO:tasks.workunit.client.0.smithi032.stdout:13 2024-09-13T12:00:15.919 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T12:00:15.919 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:00:15.919 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:00:16.572 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:00:17.573 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:00:17.573 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:00:17.573 INFO:tasks.workunit.client.0.smithi032.stdout:14 2024-09-13T12:00:17.574 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T12:00:17.574 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:00:17.574 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:00:18.294 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:00:19.295 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:00:19.295 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:00:19.295 INFO:tasks.workunit.client.0.smithi032.stdout:15 2024-09-13T12:00:19.296 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T12:00:19.296 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:00:19.296 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:00:20.047 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:00:21.048 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:00:21.048 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:00:21.048 INFO:tasks.workunit.client.0.smithi032.stdout:16 2024-09-13T12:00:21.049 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T12:00:21.049 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:00:21.049 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:00:22.153 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:00:23.154 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:00:23.155 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:00:23.155 INFO:tasks.workunit.client.0.smithi032.stdout:17 2024-09-13T12:00:23.155 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T12:00:23.155 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:00:23.155 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:00:23.688 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:00:24.690 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:00:24.690 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:00:24.690 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T12:00:24.690 INFO:tasks.workunit.client.0.smithi032.stdout:18 2024-09-13T12:00:24.691 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:00:24.691 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:00:25.144 INFO:tasks.workunit.client.0.smithi032.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1477812019,v1:127.0.0.1:6819/1477812019] [v2:127.0.0.1:6820/1477812019,v1:127.0.0.1:6821/1477812019] exists,up 4607d722-ea41-4ddb-a0cd-5570afca8326 2024-09-13T12:00:25.145 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:00:25.145 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:00:25.145 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:00:25.145 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2024-09-13T12:00:25.782 INFO:tasks.workunit.client.0.smithi032.stderr:noout is set 2024-09-13T12:00:25.795 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2024-09-13T12:00:26.605 INFO:tasks.workunit.client.0.smithi032.stderr:noin is set 2024-09-13T12:00:26.617 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2024-09-13T12:00:27.409 INFO:tasks.workunit.client.0.smithi032.stderr:nodown is set 2024-09-13T12:00:27.421 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2024-09-13T12:00:27.421 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-13T12:00:28.113 INFO:tasks.workunit.client.0.smithi032.stderr:pool 'test' already exists 2024-09-13T12:00:28.123 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T12:00:29.124 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2024-09-13T12:00:29.845 INFO:tasks.workunit.client.0.smithi032.stderr:set pool 1 size to 3 2024-09-13T12:00:29.857 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2024-09-13T12:00:30.624 INFO:tasks.workunit.client.0.smithi032.stderr:set pool 1 min_size to 2 2024-09-13T12:00:30.636 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2024-09-13T12:00:30.636 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T12:00:30.636 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T12:00:31.050 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T12:00:31.050 INFO:tasks.workunit.client.0.smithi032.stderr:1 2024-09-13T12:00:31.050 INFO:tasks.workunit.client.0.smithi032.stderr:2' 2024-09-13T12:00:31.050 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T12:00:31.050 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:00:31.051 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T12:00:31.175 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836499 2024-09-13T12:00:31.175 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836499 2024-09-13T12:00:31.175 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499' 2024-09-13T12:00:31.175 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:00:31.175 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T12:00:31.323 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672975 2024-09-13T12:00:31.324 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672975 2024-09-13T12:00:31.324 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499 1-42949672975' 2024-09-13T12:00:31.324 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:00:31.324 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T12:00:31.448 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509448 2024-09-13T12:00:31.448 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509448 2024-09-13T12:00:31.448 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499 1-42949672975 2-64424509448' 2024-09-13T12:00:31.448 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:00:31.449 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836499 2024-09-13T12:00:31.449 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:00:31.450 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T12:00:31.450 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836499 2024-09-13T12:00:31.450 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:00:31.451 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836499 2024-09-13T12:00:31.451 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-09-13T12:00:31.452 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.0 seq 21474836499 2024-09-13T12:00:31.452 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:00:31.899 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836497 -lt 21474836499 2024-09-13T12:00:31.899 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T12:00:32.900 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T12:00:32.901 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:00:33.324 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836499 -lt 21474836499 2024-09-13T12:00:33.324 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:00:33.325 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672975 2024-09-13T12:00:33.325 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:00:33.326 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T12:00:33.326 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672975 2024-09-13T12:00:33.326 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:00:33.327 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672975 2024-09-13T12:00:33.327 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-13T12:00:33.328 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.1 seq 42949672975 2024-09-13T12:00:33.328 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T12:00:33.852 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672975 -lt 42949672975 2024-09-13T12:00:33.852 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:00:33.853 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509448 2024-09-13T12:00:33.853 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:00:33.853 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T12:00:33.854 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509448 2024-09-13T12:00:33.854 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:00:33.855 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509448 2024-09-13T12:00:33.855 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-13T12:00:33.856 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.2 seq 64424509448 2024-09-13T12:00:33.856 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T12:00:34.295 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509448 -lt 64424509448 2024-09-13T12:00:34.295 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2024-09-13T12:00:34.296 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T12:00:34.296 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T12:00:34.296 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T12:00:34.296 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T12:00:34.297 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T12:00:34.297 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T12:00:34.297 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T12:00:34.297 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T12:00:34.297 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T12:00:34.386 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T12:00:34.386 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T12:00:34.386 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T12:00:34.386 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T12:00:34.387 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T12:00:34.387 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T12:00:34.838 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T12:00:34.838 INFO:tasks.workunit.client.0.smithi032.stderr:1 2024-09-13T12:00:34.838 INFO:tasks.workunit.client.0.smithi032.stderr:2' 2024-09-13T12:00:34.838 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T12:00:34.838 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:00:34.839 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T12:00:34.962 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836501 2024-09-13T12:00:34.962 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836501 2024-09-13T12:00:34.962 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501' 2024-09-13T12:00:34.963 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:00:34.963 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T12:00:35.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672978 2024-09-13T12:00:35.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672978 2024-09-13T12:00:35.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672978' 2024-09-13T12:00:35.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:00:35.087 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T12:00:35.212 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509451 2024-09-13T12:00:35.212 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509451 2024-09-13T12:00:35.212 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672978 2-64424509451' 2024-09-13T12:00:35.212 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:00:35.212 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836501 2024-09-13T12:00:35.213 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:00:35.213 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T12:00:35.214 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836501 2024-09-13T12:00:35.214 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:00:35.215 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836501 2024-09-13T12:00:35.215 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.0 seq 21474836501 2024-09-13T12:00:35.216 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-13T12:00:35.216 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:00:35.648 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836499 -lt 21474836501 2024-09-13T12:00:35.648 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T12:00:36.649 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T12:00:36.650 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:00:37.100 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836501 2024-09-13T12:00:37.101 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:00:37.101 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672978 2024-09-13T12:00:37.101 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:00:37.102 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T12:00:37.103 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672978 2024-09-13T12:00:37.103 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:00:37.104 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672978 2024-09-13T12:00:37.104 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.1 seq 42949672978 2024-09-13T12:00:37.104 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-09-13T12:00:37.104 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T12:00:37.565 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672978 -lt 42949672978 2024-09-13T12:00:37.566 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:00:37.566 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509451 2024-09-13T12:00:37.566 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:00:37.567 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T12:00:37.568 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509451 2024-09-13T12:00:37.568 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:00:37.569 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509451 2024-09-13T12:00:37.569 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.2 seq 64424509451 2024-09-13T12:00:37.569 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-13T12:00:37.569 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T12:00:38.045 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509451 -lt 64424509451 2024-09-13T12:00:38.046 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T12:00:38.046 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:00:38.046 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:00:38.602 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T12:00:38.602 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:00:38.603 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:00:38.603 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:00:38.603 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:00:38.604 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:00:38.604 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:00:38.604 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:00:39.035 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T12:00:39.035 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:00:39.036 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:00:39.036 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:00:39.593 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T12:00:39.593 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T12:00:39.593 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T12:00:39.594 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2024-09-13T12:00:39.594 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2024-09-13T12:00:39.996 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:00:40.008 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2024-09-13T12:00:40.008 INFO:tasks.workunit.client.0.smithi032.stdout:primary and soon to be divergent is 1 2024-09-13T12:00:40.008 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2024-09-13T12:00:40.008 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2024-09-13T12:00:40.399 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:00:40.399 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T12:00:29.434744+0000 0'0 28:17 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-13T12:00:27.843316+0000 0'0 2024-09-13T12:00:27.843316+0000 0 0 periodic scrub scheduled @ 2024-09-14T21:43:07.755659+0000 0 0 2024-09-13T12:00:40.399 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:00:40.399 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:00:40.400 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:00:40.411 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2024-09-13T12:00:40.411 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-09-13T12:00:40.411 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2024-09-13T12:00:40.411 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2024-09-13T12:00:40.412 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-09-13T12:00:40.412 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2024-09-13T12:00:40.412 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2024-09-13T12:00:40.412 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-09-13T12:00:40.412 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2024-09-13T12:00:40.412 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2024-09-13T12:00:40.412 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2024-09-13T12:00:40.412 INFO:tasks.workunit.client.0.smithi032.stdout:writing initial objects 2024-09-13T12:00:40.413 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2024-09-13T12:00:40.413 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:40.413 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2024-09-13T12:00:40.557 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:40.557 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2024-09-13T12:00:40.699 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:40.700 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2024-09-13T12:00:40.842 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:40.842 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2024-09-13T12:00:41.009 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:41.009 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2024-09-13T12:00:41.151 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:41.152 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2024-09-13T12:00:41.311 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:41.311 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2024-09-13T12:00:41.512 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:41.512 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2024-09-13T12:00:41.663 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:41.663 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2024-09-13T12:00:41.805 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:41.806 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2024-09-13T12:00:41.956 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:41.956 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2024-09-13T12:00:42.107 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:42.107 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2024-09-13T12:00:42.249 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:42.250 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2024-09-13T12:00:42.409 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:42.409 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2024-09-13T12:00:42.551 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:42.551 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2024-09-13T12:00:42.693 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:42.694 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2024-09-13T12:00:42.852 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:42.852 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2024-09-13T12:00:43.003 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:43.003 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2024-09-13T12:00:43.145 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:43.145 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2024-09-13T12:00:43.313 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:43.314 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2024-09-13T12:00:43.473 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:43.473 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2024-09-13T12:00:43.615 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:43.616 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2024-09-13T12:00:43.758 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:43.758 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2024-09-13T12:00:43.900 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:43.900 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2024-09-13T12:00:44.043 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:44.043 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2024-09-13T12:00:44.185 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:44.185 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2024-09-13T12:00:44.344 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:44.345 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2024-09-13T12:00:44.486 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:44.487 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2024-09-13T12:00:44.637 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:44.638 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2024-09-13T12:00:44.789 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:44.789 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2024-09-13T12:00:44.958 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:44.958 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2024-09-13T12:00:45.115 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:45.115 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2024-09-13T12:00:45.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:45.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2024-09-13T12:00:45.450 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:45.451 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2024-09-13T12:00:45.592 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:45.593 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2024-09-13T12:00:45.735 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:45.735 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2024-09-13T12:00:45.886 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:45.886 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2024-09-13T12:00:46.037 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:46.037 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2024-09-13T12:00:46.212 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:46.212 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2024-09-13T12:00:46.372 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:46.372 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2024-09-13T12:00:46.514 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:46.515 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2024-09-13T12:00:46.665 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:46.665 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2024-09-13T12:00:46.807 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:46.807 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2024-09-13T12:00:46.950 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:46.950 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2024-09-13T12:00:47.092 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:47.092 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2024-09-13T12:00:47.435 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:47.436 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2024-09-13T12:00:47.603 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:47.603 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2024-09-13T12:00:47.771 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:48.006 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2024-09-13T12:00:48.007 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:48.007 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2024-09-13T12:00:48.097 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:48.097 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2024-09-13T12:00:48.240 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:48.240 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2024-09-13T12:00:48.415 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:48.415 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2024-09-13T12:00:48.558 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:48.558 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2024-09-13T12:00:48.700 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:48.700 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2024-09-13T12:00:48.843 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:48.843 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2024-09-13T12:00:48.985 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:48.985 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2024-09-13T12:00:49.136 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:49.136 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2024-09-13T12:00:49.315 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:49.315 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2024-09-13T12:00:49.488 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:49.488 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2024-09-13T12:00:49.639 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:49.639 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2024-09-13T12:00:49.790 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:49.790 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2024-09-13T12:00:49.932 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:49.932 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2024-09-13T12:00:50.074 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:50.074 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2024-09-13T12:00:50.217 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:50.217 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2024-09-13T12:00:50.368 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:50.368 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2024-09-13T12:00:50.510 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:50.510 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2024-09-13T12:00:50.644 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:50.644 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2024-09-13T12:00:50.820 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:50.820 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2024-09-13T12:00:50.970 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:50.970 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2024-09-13T12:00:51.113 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:51.113 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2024-09-13T12:00:51.289 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:51.289 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2024-09-13T12:00:51.465 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:51.465 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2024-09-13T12:00:51.608 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:51.608 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2024-09-13T12:00:51.750 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:51.750 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2024-09-13T12:00:51.893 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:51.893 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2024-09-13T12:00:52.035 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:52.035 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2024-09-13T12:00:52.177 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:52.177 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2024-09-13T12:00:52.387 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:52.387 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2024-09-13T12:00:52.529 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:52.529 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2024-09-13T12:00:52.673 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:52.673 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2024-09-13T12:00:52.831 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:52.831 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2024-09-13T12:00:52.973 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:52.974 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2024-09-13T12:00:53.116 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:53.116 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2024-09-13T12:00:53.275 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:53.275 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2024-09-13T12:00:53.468 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:53.468 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2024-09-13T12:00:53.618 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:53.619 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2024-09-13T12:00:53.761 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:53.761 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2024-09-13T12:00:53.911 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:53.912 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2024-09-13T12:00:54.079 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:54.079 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2024-09-13T12:00:54.221 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:54.222 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2024-09-13T12:00:54.389 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:54.390 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2024-09-13T12:00:54.531 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:54.532 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2024-09-13T12:00:54.682 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:54.683 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2024-09-13T12:00:54.833 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:54.834 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2024-09-13T12:00:54.975 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:54.976 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2024-09-13T12:00:55.121 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:55.121 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2024-09-13T12:00:55.269 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:55.269 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2024-09-13T12:00:55.445 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:55.445 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2024-09-13T12:00:55.587 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:55.587 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2024-09-13T12:00:55.732 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-13T12:00:55.733 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-09-13T12:00:55.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-13T12:00:55.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2024-09-13T12:00:55.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T12:00:55.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T12:00:55.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T12:00:55.924 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-13T12:00:55.924 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T12:00:55.924 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T12:00:55.924 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T12:00:55.924 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T12:00:55.924 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T12:00:55.980 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-13T12:00:55.980 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T12:00:55.980 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T12:00:55.980 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T12:00:55.980 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T12:00:55.980 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T12:00:56.440 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T12:00:56.440 INFO:tasks.workunit.client.0.smithi032.stderr:1 2024-09-13T12:00:56.441 INFO:tasks.workunit.client.0.smithi032.stderr:2' 2024-09-13T12:00:56.441 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T12:00:56.441 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:00:56.441 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T12:00:56.593 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836508 2024-09-13T12:00:56.593 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836508 2024-09-13T12:00:56.593 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508' 2024-09-13T12:00:56.593 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:00:56.594 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T12:00:56.717 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672984 2024-09-13T12:00:56.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672984 2024-09-13T12:00:56.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508 1-42949672984' 2024-09-13T12:00:56.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:00:56.718 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T12:00:56.843 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509457 2024-09-13T12:00:56.843 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509457 2024-09-13T12:00:56.843 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508 1-42949672984 2-64424509457' 2024-09-13T12:00:56.843 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:00:56.843 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836508 2024-09-13T12:00:56.844 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:00:56.844 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T12:00:56.845 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836508 2024-09-13T12:00:56.845 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:00:56.846 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836508 2024-09-13T12:00:56.846 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-09-13T12:00:56.846 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.0 seq 21474836508 2024-09-13T12:00:56.847 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:00:57.293 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836508 2024-09-13T12:00:57.293 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T12:00:58.295 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T12:00:58.296 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:00:58.744 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836508 -lt 21474836508 2024-09-13T12:00:58.745 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:00:58.745 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672984 2024-09-13T12:00:58.745 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:00:58.746 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T12:00:58.747 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672984 2024-09-13T12:00:58.747 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:00:58.748 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672984 2024-09-13T12:00:58.748 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-09-13T12:00:58.748 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.1 seq 42949672984 2024-09-13T12:00:58.748 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T12:00:59.196 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672984 -lt 42949672984 2024-09-13T12:00:59.196 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:00:59.196 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509457 2024-09-13T12:00:59.197 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:00:59.197 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T12:00:59.198 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509457 2024-09-13T12:00:59.198 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:00:59.199 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509457 2024-09-13T12:00:59.199 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-09-13T12:00:59.199 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.2 seq 64424509457 2024-09-13T12:00:59.200 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T12:00:59.680 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509457 -lt 64424509457 2024-09-13T12:00:59.681 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T12:00:59.681 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:00:59.681 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:01:00.233 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T12:01:00.233 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:01:00.233 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:01:00.233 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:01:00.234 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:01:00.234 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:01:00.234 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:01:00.234 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:01:00.675 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T12:01:00.676 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:01:00.676 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:01:00.676 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:01:01.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T12:01:01.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T12:01:01.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T12:01:01.234 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2024-09-13T12:01:01.234 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=test 2024-09-13T12:01:01.235 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=existing_1 2024-09-13T12:01:01.235 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map test existing_1 2024-09-13T12:01:01.235 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T12:01:01.701 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2024-09-13T12:01:01.702 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2024-09-13T12:01:01.702 INFO:tasks.workunit.client.0.smithi032.stdout:blackholing osds 0 2 2024-09-13T12:01:01.702 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2024-09-13T12:01:02.097 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:01:02.097 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 100 0 0 0 0 48300 0 0 16 84 16 active+clean 2024-09-13T12:00:29.434744+0000 28'100 28:117 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-13T12:00:27.843316+0000 0'0 2024-09-13T12:00:27.843316+0000 0 0 periodic scrub scheduled @ 2024-09-14T21:43:07.755659+0000 0 0 2024-09-13T12:01:02.097 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:01:02.097 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:01:02.097 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:01:02.108 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-09-13T12:01:02.109 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2024-09-13T12:01:02.109 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-13T12:01:02.109 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-13T12:01:02.110 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:01:02.110 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:01:02.110 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:01:02.110 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-osd.0.asok 2024-09-13T12:01:02.110 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-09-13T12:01:02.110 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.33242/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-13T12:01:02.189 INFO:tasks.workunit.client.0.smithi032.stdout:{ 2024-09-13T12:01:02.190 INFO:tasks.workunit.client.0.smithi032.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-13T12:01:02.190 INFO:tasks.workunit.client.0.smithi032.stdout:} 2024-09-13T12:01:02.196 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-09-13T12:01:02.198 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2024-09-13T12:01:02.198 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-13T12:01:02.198 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-13T12:01:02.198 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:01:02.198 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:01:02.198 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:01:02.198 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-osd.2.asok 2024-09-13T12:01:02.198 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-09-13T12:01:02.199 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.33242/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-13T12:01:02.278 INFO:tasks.workunit.client.0.smithi032.stdout:{ 2024-09-13T12:01:02.279 INFO:tasks.workunit.client.0.smithi032.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-13T12:01:02.279 INFO:tasks.workunit.client.0.smithi032.stdout:} 2024-09-13T12:01:02.285 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2024-09-13T12:01:02.286 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2024-09-13T12:01:02.287 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2024-09-13T12:01:02.287 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2024-09-13T12:01:02.288 INFO:tasks.workunit.client.0.smithi032.stdout:writing divergent object 2024-09-13T12:01:02.288 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2024-09-13T12:01:02.288 INFO:tasks.workunit.client.0.smithi032.stdout:create missing divergent object 2024-09-13T12:01:02.290 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2024-09-13T12:01:02.290 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-09-13T12:01:02.290 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-13T12:01:02.290 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-13T12:01:02.290 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-13T12:01:02.290 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-13T12:01:02.291 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=test 2024-09-13T12:01:02.291 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-13T12:01:02.291 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=existing_99 2024-09-13T12:01:02.291 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-13T12:01:02.291 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/divergent-priors 2024-09-13T12:01:02.291 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-13T12:01:02.291 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-13T12:01:02.291 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-13T12:01:02.291 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds test existing_99 2024-09-13T12:01:02.291 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=test 2024-09-13T12:01:02.291 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=existing_99 2024-09-13T12:01:02.291 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map test existing_99 2024-09-13T12:01:02.291 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-13T12:01:02.723 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-13T12:01:02.724 INFO:tasks.workunit.client.0.smithi032.stderr:0 2024-09-13T12:01:02.724 INFO:tasks.workunit.client.0.smithi032.stderr:2' 2024-09-13T12:01:02.724 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2 2024-09-13T12:01:02.724 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0' '2') 2024-09-13T12:01:02.724 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-13T12:01:02.724 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-13T12:01:02.724 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-13T12:01:02.724 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-13T12:01:02.724 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/divergent-priors/1/type 2024-09-13T12:01:02.726 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-13T12:01:02.726 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-13T12:01:02.726 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-13T12:01:02.726 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-13T12:01:02.726 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-13T12:01:02.726 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-13T12:01:02.726 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-13T12:01:02.726 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-13T12:01:02.726 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-13T12:01:02.726 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-13T12:01:02.727 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:01:02.727 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:01:02.727 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:01:02.727 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-osd.1.asok 2024-09-13T12:01:02.727 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.33242/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-13T12:01:02.815 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-13T12:01:02.815 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-13T12:01:02.816 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-13T12:01:02.816 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-13T12:01:02.817 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-13T12:01:02.817 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-13T12:01:02.817 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:01:02.817 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:01:02.817 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:01:02.817 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-osd.1.asok 2024-09-13T12:01:02.817 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-13T12:01:02.817 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.33242/ceph-osd.1.asok injectdataerr test existing_99 2024-09-13T12:01:02.906 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2024-09-13T12:01:02.906 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2024-09-13T12:01:12.907 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2024-09-13T12:01:12.919 INFO:tasks.workunit.client.0.smithi032.stderr:rados: no process found 2024-09-13T12:01:12.920 INFO:tasks.workunit.client.0.smithi032.stdout:killing all the osds 2024-09-13T12:01:12.920 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2024-09-13T12:01:12.920 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2024-09-13T12:01:13.326 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:01:13.326 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 100 0 0 0 0 48300 0 0 17 84 17 active+clean 2024-09-13T12:01:03.083628+0000 28'101 28:126 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-13T12:00:27.843316+0000 0'0 2024-09-13T12:00:27.843316+0000 0 0 periodic scrub scheduled @ 2024-09-14T21:43:07.755659+0000 0 0 2024-09-13T12:01:13.326 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:01:13.326 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:01:13.326 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:01:13.338 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2024-09-13T12:01:13.339 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:01:13.339 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:01:13.339 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:01:13.339 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:01:13.339 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:01:13.445 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:01:13.445 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-09-13T12:01:13.445 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2024-09-13T12:01:13.959 INFO:tasks.workunit.client.0.smithi032.stderr:osd.0 is already down. 2024-09-13T12:01:13.971 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-09-13T12:01:13.971 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2024-09-13T12:01:14.576 INFO:tasks.workunit.client.0.smithi032.stderr:osd.1 is already down. 2024-09-13T12:01:14.588 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-09-13T12:01:14.588 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2024-09-13T12:01:15.106 INFO:tasks.workunit.client.0.smithi032.stderr:osd.2 is already down. 2024-09-13T12:01:15.117 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-09-13T12:01:15.117 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2024-09-13T12:01:15.737 INFO:tasks.workunit.client.0.smithi032.stderr:osd.0 is already out. 2024-09-13T12:01:15.749 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-09-13T12:01:15.749 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2024-09-13T12:01:16.272 INFO:tasks.workunit.client.0.smithi032.stderr:osd.2 is already out. 2024-09-13T12:01:16.283 INFO:tasks.workunit.client.0.smithi032.stdout:bringing up non_divergent 0 2 2024-09-13T12:01:16.284 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2024-09-13T12:01:16.284 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2024-09-13T12:01:16.701 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:01:16.702 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 100 0 0 0 0 48300 0 0 17 84 17 stale+active+clean 2024-09-13T12:01:03.083628+0000 28'101 28:126 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-13T12:00:27.843316+0000 0'0 2024-09-13T12:00:27.843316+0000 0 0 periodic scrub scheduled @ 2024-09-14T21:43:07.755659+0000 0 0 2024-09-13T12:01:16.702 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:01:16.702 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:01:16.702 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:01:16.716 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-09-13T12:01:16.716 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2024-09-13T12:01:16.716 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-13T12:01:16.716 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:01:16.716 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T12:01:16.716 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:01:16.716 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/0 2024-09-13T12:01:16.716 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T12:01:16.716 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:01:16.717 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:01:16.717 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:01:16.717 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-13T12:01:16.717 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-13T12:01:16.717 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:01:16.717 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:01:16.717 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T12:01:16.717 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:01:16.717 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:01:16.717 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:01:16.717 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:01:16.717 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:01:16.717 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:01:16.718 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:01:16.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:01:16.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:01:16.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T12:01:16.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T12:01:16.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:01:16.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:01:16.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:01:16.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:01:16.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:01:16.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:01:16.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/0 2024-09-13T12:01:16.719 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.0 2024-09-13T12:01:16.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T12:01:16.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:01:16.720 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/0/whoami 2024-09-13T12:01:16.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T12:01:16.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:01:16.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:01:16.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:01:16.752 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:01:16.751+0000 7fed84c16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:01:16.754 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:01:16.753+0000 7fed84c16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:01:16.755 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:01:16.754+0000 7fed84c16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:01:17.145 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T12:01:17.146 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:01:17.146 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T12:01:17.146 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:01:17.146 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:01:17.146 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:01:17.146 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:01:17.147 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:01:17.147 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:01:17.147 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:01:17.588 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:01:18.333 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:01:18.332+0000 7fed84c16640 -1 Falling back to public interface 2024-09-13T12:01:18.590 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:01:18.590 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:01:18.590 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:01:18.590 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:01:18.591 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:01:18.591 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:01:19.039 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:01:20.041 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:01:20.041 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:01:20.041 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:01:20.041 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:01:20.041 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:01:20.042 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:01:20.486 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:01:21.367 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:01:21.366+0000 7fed84c16640 -1 osd.0 28 log_to_monitors true 2024-09-13T12:01:21.487 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:01:21.487 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:01:21.487 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:01:21.488 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:01:21.488 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:01:21.488 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:01:21.927 INFO:tasks.workunit.client.0.smithi032.stdout:osd.0 up out weight 0 up_from 35 up_thru 0 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/442043230,v1:127.0.0.1:6803/442043230] [v2:127.0.0.1:6804/442043230,v1:127.0.0.1:6805/442043230] exists,up 42e90f08-426a-4b5b-ad78-7bd0b8144f62 2024-09-13T12:01:21.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:01:21.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:01:21.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:01:21.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-09-13T12:01:21.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2024-09-13T12:01:21.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-13T12:01:21.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:01:21.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-13T12:01:21.932 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:01:21.932 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/2 2024-09-13T12:01:21.932 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T12:01:21.932 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:01:21.932 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:01:21.932 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:01:21.932 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-13T12:01:21.932 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-13T12:01:21.932 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:01:21.932 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:01:21.932 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T12:01:21.932 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:01:21.932 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:01:21.933 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:01:21.933 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:01:21.933 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:01:21.933 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:01:21.933 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:01:21.933 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:01:21.933 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:01:21.933 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T12:01:21.933 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T12:01:21.933 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:01:21.934 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:01:21.934 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:01:21.934 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:01:21.934 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:01:21.934 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:01:21.934 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/2 2024-09-13T12:01:21.934 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.2 2024-09-13T12:01:21.935 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-13T12:01:21.935 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:01:21.935 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/2/whoami 2024-09-13T12:01:21.935 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-13T12:01:21.935 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:01:21.935 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:01:21.935 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:01:21.965 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:01:21.964+0000 7f782ed02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:01:21.967 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:01:21.966+0000 7f782ed02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:01:21.968 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:01:21.967+0000 7f782ed02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:01:22.384 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-13T12:01:22.384 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:01:22.384 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T12:01:22.384 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:01:22.384 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:01:22.384 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:01:22.385 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:01:22.385 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:01:22.385 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:01:22.385 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:01:22.826 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:01:23.072 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:01:23.071+0000 7f782ed02640 -1 Falling back to public interface 2024-09-13T12:01:23.827 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:01:23.827 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:01:23.828 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:01:23.828 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:01:23.828 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:01:23.828 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:01:24.278 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:01:25.280 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:01:25.281 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:01:25.281 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:01:25.282 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:01:25.282 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:01:25.282 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:01:25.691 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:01:26.239 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:01:26.238+0000 7f782ed02640 -1 osd.2 28 log_to_monitors true 2024-09-13T12:01:26.693 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:01:26.693 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:01:26.693 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:01:26.694 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:01:26.694 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:01:26.694 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:01:27.146 INFO:tasks.workunit.client.0.smithi032.stdout:osd.2 up out weight 0 up_from 37 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/3448235176,v1:127.0.0.1:6811/3448235176] [v2:127.0.0.1:6812/3448235176,v1:127.0.0.1:6813/3448235176] exists,up 4607d722-ea41-4ddb-a0cd-5570afca8326 2024-09-13T12:01:27.147 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:01:27.147 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:01:27.147 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:01:27.147 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-09-13T12:01:27.147 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2024-09-13T12:01:27.662 INFO:tasks.workunit.client.0.smithi032.stderr:osd.0 is already in. 2024-09-13T12:01:27.674 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-09-13T12:01:27.674 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2024-09-13T12:01:28.246 INFO:tasks.workunit.client.0.smithi032.stderr:osd.2 is already in. 2024-09-13T12:01:28.259 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-13T12:01:28.259 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2024-09-13T12:01:28.259 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T12:01:28.259 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T12:01:28.259 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T12:01:28.259 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-13T12:01:28.260 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T12:01:28.260 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T12:01:28.260 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T12:01:28.260 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T12:01:28.260 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T12:01:28.314 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-13T12:01:28.314 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T12:01:28.314 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T12:01:28.314 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T12:01:28.315 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T12:01:28.315 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T12:01:28.767 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T12:01:28.768 INFO:tasks.workunit.client.0.smithi032.stderr:1 2024-09-13T12:01:28.768 INFO:tasks.workunit.client.0.smithi032.stderr:2' 2024-09-13T12:01:28.768 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T12:01:28.768 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:01:28.768 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T12:01:28.967 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855364 2024-09-13T12:01:28.967 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855364 2024-09-13T12:01:28.967 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855364' 2024-09-13T12:01:28.968 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:01:28.968 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T12:01:29.079 INFO:tasks.workunit.client.0.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-13T12:01:29.080 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-13T12:01:29.081 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-13T12:01:29.081 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-13T12:01:29.081 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:01:29.081 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T12:01:29.206 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=158913789955 2024-09-13T12:01:29.206 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 158913789955 2024-09-13T12:01:29.206 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855364 2-158913789955' 2024-09-13T12:01:29.206 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:01:29.206 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-150323855364 2024-09-13T12:01:29.206 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:01:29.207 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T12:01:29.207 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-150323855364 2024-09-13T12:01:29.207 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:01:29.208 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855364 2024-09-13T12:01:29.208 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 150323855364' 2024-09-13T12:01:29.209 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.0 seq 150323855364 2024-09-13T12:01:29.209 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:01:29.671 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855362 -lt 150323855364 2024-09-13T12:01:29.672 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T12:01:30.673 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T12:01:30.673 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:01:31.132 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855364 -lt 150323855364 2024-09-13T12:01:31.132 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:01:31.133 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-158913789955 2024-09-13T12:01:31.133 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:01:31.133 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T12:01:31.134 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-158913789955 2024-09-13T12:01:31.134 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:01:31.135 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=158913789955 2024-09-13T12:01:31.135 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 158913789955' 2024-09-13T12:01:31.135 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.2 seq 158913789955 2024-09-13T12:01:31.136 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T12:01:31.583 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 158913789955 -lt 158913789955 2024-09-13T12:01:31.584 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T12:01:31.584 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:01:31.584 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:01:32.138 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T12:01:32.138 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:01:32.138 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:01:32.138 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:01:32.139 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:01:32.139 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:01:32.139 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:01:32.139 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:01:32.577 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:01:32.580 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:01:32.580 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:01:32.580 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:01:33.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:01:33.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-13T12:01:33.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T12:01:33.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-13T12:01:33.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:01:33.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T12:01:33.240 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:01:33.240 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:01:33.240 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:01:33.240 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:01:33.240 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:01:33.241 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:01:33.241 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:01:33.241 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:01:33.653 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:01:33.653 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:01:33.654 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:01:33.654 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:01:34.217 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:01:34.217 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:01:34.217 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:01:34.217 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:01:34.217 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:01:34.217 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:01:34.218 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:01:34.218 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:01:34.218 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:01:34.770 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:01:34.770 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:01:34.770 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-13T12:01:34.770 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:01:34.770 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T12:01:34.971 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:01:34.971 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:01:34.971 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:01:34.972 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:01:34.972 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:01:34.972 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:01:34.972 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:01:34.972 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:01:35.377 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:01:35.377 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:01:35.377 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:01:35.377 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:01:35.926 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:01:35.926 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:01:35.926 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:01:35.926 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:01:35.927 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:01:35.927 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:01:35.927 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:01:35.927 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:01:35.927 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:01:36.526 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:01:36.526 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:01:36.526 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-13T12:01:36.526 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:01:36.526 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-13T12:01:36.927 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:01:36.927 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:01:36.928 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:01:36.928 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:01:36.928 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:01:36.928 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:01:36.928 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:01:36.928 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:01:37.332 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:01:37.332 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:01:37.332 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:01:37.333 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:01:37.887 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:01:37.887 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:01:37.887 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:01:37.887 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:01:37.887 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:01:37.887 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:01:37.887 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:01:37.887 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:01:37.887 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:01:38.453 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:01:38.453 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:01:38.453 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-13T12:01:38.453 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:01:38.453 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-13T12:01:39.254 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:01:39.254 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:01:39.255 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:01:39.255 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:01:39.255 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:01:39.255 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:01:39.256 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:01:39.256 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:01:39.671 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:01:39.671 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:01:39.671 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:01:39.671 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:01:40.229 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:01:40.230 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:01:40.230 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:01:40.230 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:01:40.230 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:01:40.230 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:01:40.230 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:01:40.230 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:01:40.230 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:01:40.791 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:01:40.791 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:01:40.791 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-13T12:01:40.791 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:01:40.791 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-13T12:01:42.392 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:01:42.392 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:01:42.393 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:01:42.393 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:01:42.393 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:01:42.393 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:01:42.393 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:01:42.393 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:01:42.810 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:01:42.811 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:01:42.811 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:01:42.811 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:01:43.372 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:01:43.372 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:01:43.372 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:01:43.372 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:01:43.372 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:01:43.372 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:01:43.372 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:01:43.373 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:01:43.373 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:01:43.929 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:01:43.929 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:01:43.929 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-13T12:01:43.929 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:01:43.929 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-13T12:01:47.130 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:01:47.130 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:01:47.130 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:01:47.131 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:01:47.131 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:01:47.131 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:01:47.131 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:01:47.131 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:01:47.569 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:01:47.569 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:01:47.569 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:01:47.569 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:01:48.125 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:01:48.125 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:01:48.125 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:01:48.125 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:01:48.125 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:01:48.125 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:01:48.126 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:01:48.126 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:01:48.126 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:01:48.703 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:01:48.703 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:01:48.703 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-13T12:01:48.703 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:01:48.703 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-13T12:01:55.104 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:01:55.104 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:01:55.105 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:01:55.105 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:01:55.105 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:01:55.105 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:01:55.105 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:01:55.105 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:01:55.520 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:01:55.520 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:01:55.521 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:01:55.521 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:01:56.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:01:56.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:01:56.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:01:56.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:01:56.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:01:56.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:01:56.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:01:56.087 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:01:56.088 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:01:56.661 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:01:56.661 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:01:56.662 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-13T12:01:56.662 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:01:56.662 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-13T12:02:03.962 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:02:03.963 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:02:03.963 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:02:03.963 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:02:03.964 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:02:03.964 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:02:03.964 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:02:03.964 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:02:04.376 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:02:04.376 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:02:04.376 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:02:04.376 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:02:04.926 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:02:04.926 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:02:04.926 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:02:04.926 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:02:04.926 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:02:04.926 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:02:04.926 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:02:04.926 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:02:04.926 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:02:05.515 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:02:05.515 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:02:05.515 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-13T12:02:05.515 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-13T12:02:06.064 INFO:tasks.workunit.client.0.smithi032.stdout:{ 2024-09-13T12:02:06.064 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_fingerprint": "74963695-4483-4f33-bb22-b0f90d566d06", 2024-09-13T12:02:06.064 INFO:tasks.workunit.client.0.smithi032.stdout: "version": "19.3.0-4697-gd58f7306", 2024-09-13T12:02:06.064 INFO:tasks.workunit.client.0.smithi032.stdout: "commit": "d58f73069a8157a9ace12a0247ee3e8497707ea9", 2024-09-13T12:02:06.064 INFO:tasks.workunit.client.0.smithi032.stdout: "timestamp": "2024-09-13T12:02:06.062226+0000", 2024-09-13T12:02:06.064 INFO:tasks.workunit.client.0.smithi032.stdout: "tag": "", 2024-09-13T12:02:06.064 INFO:tasks.workunit.client.0.smithi032.stdout: "health": { 2024-09-13T12:02:06.064 INFO:tasks.workunit.client.0.smithi032.stdout: "status": "HEALTH_WARN", 2024-09-13T12:02:06.064 INFO:tasks.workunit.client.0.smithi032.stdout: "checks": { 2024-09-13T12:02:06.064 INFO:tasks.workunit.client.0.smithi032.stdout: "OSDMAP_FLAGS": { 2024-09-13T12:02:06.065 INFO:tasks.workunit.client.0.smithi032.stdout: "severity": "HEALTH_WARN", 2024-09-13T12:02:06.065 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": { 2024-09-13T12:02:06.065 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-13T12:02:06.065 INFO:tasks.workunit.client.0.smithi032.stdout: "count": 17 2024-09-13T12:02:06.065 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.065 INFO:tasks.workunit.client.0.smithi032.stdout: "detail": [], 2024-09-13T12:02:06.065 INFO:tasks.workunit.client.0.smithi032.stdout: "muted": false 2024-09-13T12:02:06.065 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.065 INFO:tasks.workunit.client.0.smithi032.stdout: "OSD_DOWN": { 2024-09-13T12:02:06.065 INFO:tasks.workunit.client.0.smithi032.stdout: "severity": "HEALTH_WARN", 2024-09-13T12:02:06.065 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": { 2024-09-13T12:02:06.065 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "1 osds down", 2024-09-13T12:02:06.066 INFO:tasks.workunit.client.0.smithi032.stdout: "count": 1 2024-09-13T12:02:06.066 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.066 INFO:tasks.workunit.client.0.smithi032.stdout: "detail": [ 2024-09-13T12:02:06.066 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.066 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "osd.1 (root=default,host=smithi032) is down" 2024-09-13T12:02:06.066 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.066 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:02:06.066 INFO:tasks.workunit.client.0.smithi032.stdout: "muted": false 2024-09-13T12:02:06.066 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.066 INFO:tasks.workunit.client.0.smithi032.stdout: "PG_DEGRADED": { 2024-09-13T12:02:06.066 INFO:tasks.workunit.client.0.smithi032.stdout: "severity": "HEALTH_WARN", 2024-09-13T12:02:06.066 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": { 2024-09-13T12:02:06.066 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-09-13T12:02:06.066 INFO:tasks.workunit.client.0.smithi032.stdout: "count": 1 2024-09-13T12:02:06.067 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.067 INFO:tasks.workunit.client.0.smithi032.stdout: "detail": [ 2024-09-13T12:02:06.067 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.067 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-09-13T12:02:06.067 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.067 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:02:06.067 INFO:tasks.workunit.client.0.smithi032.stdout: "muted": false 2024-09-13T12:02:06.067 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.067 INFO:tasks.workunit.client.0.smithi032.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-13T12:02:06.067 INFO:tasks.workunit.client.0.smithi032.stdout: "severity": "HEALTH_WARN", 2024-09-13T12:02:06.067 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": { 2024-09-13T12:02:06.067 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-13T12:02:06.068 INFO:tasks.workunit.client.0.smithi032.stdout: "count": 1 2024-09-13T12:02:06.068 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.068 INFO:tasks.workunit.client.0.smithi032.stdout: "detail": [ 2024-09-13T12:02:06.068 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.068 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "application not enabled on pool 'test'" 2024-09-13T12:02:06.068 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.068 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.068 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-13T12:02:06.068 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.068 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:02:06.068 INFO:tasks.workunit.client.0.smithi032.stdout: "muted": false 2024-09-13T12:02:06.068 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.068 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.068 INFO:tasks.workunit.client.0.smithi032.stdout: "mutes": [] 2024-09-13T12:02:06.069 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.069 INFO:tasks.workunit.client.0.smithi032.stdout: "monmap_first_committed": 1, 2024-09-13T12:02:06.069 INFO:tasks.workunit.client.0.smithi032.stdout: "monmap_last_committed": 1, 2024-09-13T12:02:06.069 INFO:tasks.workunit.client.0.smithi032.stdout: "monmap": { 2024-09-13T12:02:06.069 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 1, 2024-09-13T12:02:06.069 INFO:tasks.workunit.client.0.smithi032.stdout: "fsid": "e12dfbc3-9bf5-45db-a6f0-f25db93ee351", 2024-09-13T12:02:06.069 INFO:tasks.workunit.client.0.smithi032.stdout: "modified": "2024-09-13T11:58:40.423216Z", 2024-09-13T12:02:06.069 INFO:tasks.workunit.client.0.smithi032.stdout: "created": "2024-09-13T11:58:40.423216Z", 2024-09-13T12:02:06.069 INFO:tasks.workunit.client.0.smithi032.stdout: "min_mon_release": 19, 2024-09-13T12:02:06.069 INFO:tasks.workunit.client.0.smithi032.stdout: "min_mon_release_name": "squid", 2024-09-13T12:02:06.069 INFO:tasks.workunit.client.0.smithi032.stdout: "election_strategy": 1, 2024-09-13T12:02:06.069 INFO:tasks.workunit.client.0.smithi032.stdout: "disallowed_leaders: ": "", 2024-09-13T12:02:06.069 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_mode": false, 2024-09-13T12:02:06.069 INFO:tasks.workunit.client.0.smithi032.stdout: "tiebreaker_mon": "", 2024-09-13T12:02:06.070 INFO:tasks.workunit.client.0.smithi032.stdout: "removed_ranks: ": "", 2024-09-13T12:02:06.070 INFO:tasks.workunit.client.0.smithi032.stdout: "features": { 2024-09-13T12:02:06.070 INFO:tasks.workunit.client.0.smithi032.stdout: "persistent": [ 2024-09-13T12:02:06.070 INFO:tasks.workunit.client.0.smithi032.stdout: "kraken", 2024-09-13T12:02:06.070 INFO:tasks.workunit.client.0.smithi032.stdout: "luminous", 2024-09-13T12:02:06.070 INFO:tasks.workunit.client.0.smithi032.stdout: "mimic", 2024-09-13T12:02:06.070 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap-prune", 2024-09-13T12:02:06.070 INFO:tasks.workunit.client.0.smithi032.stdout: "nautilus", 2024-09-13T12:02:06.070 INFO:tasks.workunit.client.0.smithi032.stdout: "octopus", 2024-09-13T12:02:06.070 INFO:tasks.workunit.client.0.smithi032.stdout: "pacific", 2024-09-13T12:02:06.070 INFO:tasks.workunit.client.0.smithi032.stdout: "elector-pinging", 2024-09-13T12:02:06.070 INFO:tasks.workunit.client.0.smithi032.stdout: "quincy", 2024-09-13T12:02:06.070 INFO:tasks.workunit.client.0.smithi032.stdout: "reef", 2024-09-13T12:02:06.070 INFO:tasks.workunit.client.0.smithi032.stdout: "squid" 2024-09-13T12:02:06.070 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:02:06.071 INFO:tasks.workunit.client.0.smithi032.stdout: "optional": [] 2024-09-13T12:02:06.071 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.071 INFO:tasks.workunit.client.0.smithi032.stdout: "mons": [ 2024-09-13T12:02:06.071 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.071 INFO:tasks.workunit.client.0.smithi032.stdout: "rank": 0, 2024-09-13T12:02:06.071 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "a", 2024-09-13T12:02:06.071 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addrs": { 2024-09-13T12:02:06.071 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:02:06.071 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.071 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:02:06.071 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:7115", 2024-09-13T12:02:06.071 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 0 2024-09-13T12:02:06.071 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.071 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:02:06.071 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.072 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:7115/0", 2024-09-13T12:02:06.072 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-13T12:02:06.072 INFO:tasks.workunit.client.0.smithi032.stdout: "priority": 0, 2024-09-13T12:02:06.072 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 0, 2024-09-13T12:02:06.072 INFO:tasks.workunit.client.0.smithi032.stdout: "crush_location": "{}" 2024-09-13T12:02:06.072 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.072 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:02:06.072 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.072 INFO:tasks.workunit.client.0.smithi032.stdout: "quorum": [ 2024-09-13T12:02:06.072 INFO:tasks.workunit.client.0.smithi032.stdout: 0 2024-09-13T12:02:06.072 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:02:06.072 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap": { 2024-09-13T12:02:06.073 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 40, 2024-09-13T12:02:06.073 INFO:tasks.workunit.client.0.smithi032.stdout: "fsid": "e12dfbc3-9bf5-45db-a6f0-f25db93ee351", 2024-09-13T12:02:06.073 INFO:tasks.workunit.client.0.smithi032.stdout: "created": "2024-09-13T11:58:41.325052+0000", 2024-09-13T12:02:06.073 INFO:tasks.workunit.client.0.smithi032.stdout: "modified": "2024-09-13T12:01:28.528887+0000", 2024-09-13T12:02:06.073 INFO:tasks.workunit.client.0.smithi032.stdout: "last_up_change": "2024-09-13T12:01:26.438129+0000", 2024-09-13T12:02:06.073 INFO:tasks.workunit.client.0.smithi032.stdout: "last_in_change": "2024-09-13T12:01:28.024476+0000", 2024-09-13T12:02:06.073 INFO:tasks.workunit.client.0.smithi032.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T12:02:06.073 INFO:tasks.workunit.client.0.smithi032.stdout: "flags_num": 5800384, 2024-09-13T12:02:06.073 INFO:tasks.workunit.client.0.smithi032.stdout: "flags_set": [ 2024-09-13T12:02:06.073 INFO:tasks.workunit.client.0.smithi032.stdout: "nodown", 2024-09-13T12:02:06.073 INFO:tasks.workunit.client.0.smithi032.stdout: "noin", 2024-09-13T12:02:06.073 INFO:tasks.workunit.client.0.smithi032.stdout: "noout", 2024-09-13T12:02:06.073 INFO:tasks.workunit.client.0.smithi032.stdout: "pglog_hardlimit", 2024-09-13T12:02:06.074 INFO:tasks.workunit.client.0.smithi032.stdout: "purged_snapdirs", 2024-09-13T12:02:06.074 INFO:tasks.workunit.client.0.smithi032.stdout: "recovery_deletes", 2024-09-13T12:02:06.074 INFO:tasks.workunit.client.0.smithi032.stdout: "sortbitwise" 2024-09-13T12:02:06.074 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:02:06.074 INFO:tasks.workunit.client.0.smithi032.stdout: "crush_version": 9, 2024-09-13T12:02:06.074 INFO:tasks.workunit.client.0.smithi032.stdout: "full_ratio": 0.99000000953674316, 2024-09-13T12:02:06.074 INFO:tasks.workunit.client.0.smithi032.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-13T12:02:06.074 INFO:tasks.workunit.client.0.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T12:02:06.074 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_snapshot": "", 2024-09-13T12:02:06.074 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_max": 1, 2024-09-13T12:02:06.074 INFO:tasks.workunit.client.0.smithi032.stdout: "max_osd": 3, 2024-09-13T12:02:06.074 INFO:tasks.workunit.client.0.smithi032.stdout: "require_min_compat_client": "luminous", 2024-09-13T12:02:06.075 INFO:tasks.workunit.client.0.smithi032.stdout: "min_compat_client": "jewel", 2024-09-13T12:02:06.075 INFO:tasks.workunit.client.0.smithi032.stdout: "require_osd_release": "squid", 2024-09-13T12:02:06.075 INFO:tasks.workunit.client.0.smithi032.stdout: "allow_crimson": false, 2024-09-13T12:02:06.075 INFO:tasks.workunit.client.0.smithi032.stdout: "pools": [ 2024-09-13T12:02:06.075 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.075 INFO:tasks.workunit.client.0.smithi032.stdout: "pool": 1, 2024-09-13T12:02:06.075 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_name": "test", 2024-09-13T12:02:06.075 INFO:tasks.workunit.client.0.smithi032.stdout: "create_time": "2024-09-13T12:00:27.747734+0000", 2024-09-13T12:02:06.075 INFO:tasks.workunit.client.0.smithi032.stdout: "flags": 1, 2024-09-13T12:02:06.075 INFO:tasks.workunit.client.0.smithi032.stdout: "flags_names": "hashpspool", 2024-09-13T12:02:06.075 INFO:tasks.workunit.client.0.smithi032.stdout: "type": 1, 2024-09-13T12:02:06.075 INFO:tasks.workunit.client.0.smithi032.stdout: "size": 3, 2024-09-13T12:02:06.075 INFO:tasks.workunit.client.0.smithi032.stdout: "min_size": 2, 2024-09-13T12:02:06.075 INFO:tasks.workunit.client.0.smithi032.stdout: "crush_rule": 0, 2024-09-13T12:02:06.076 INFO:tasks.workunit.client.0.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-13T12:02:06.076 INFO:tasks.workunit.client.0.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-13T12:02:06.076 INFO:tasks.workunit.client.0.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T12:02:06.076 INFO:tasks.workunit.client.0.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T12:02:06.076 INFO:tasks.workunit.client.0.smithi032.stdout: "is_stretch_pool": false, 2024-09-13T12:02:06.076 INFO:tasks.workunit.client.0.smithi032.stdout: "object_hash": 2, 2024-09-13T12:02:06.076 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_autoscale_mode": "off", 2024-09-13T12:02:06.076 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_num": 1, 2024-09-13T12:02:06.076 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_placement_num": 1, 2024-09-13T12:02:06.076 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_placement_num_target": 1, 2024-09-13T12:02:06.076 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_num_target": 1, 2024-09-13T12:02:06.076 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_num_pending": 1, 2024-09-13T12:02:06.076 INFO:tasks.workunit.client.0.smithi032.stdout: "last_pg_merge_meta": { 2024-09-13T12:02:06.077 INFO:tasks.workunit.client.0.smithi032.stdout: "source_pgid": "0.0", 2024-09-13T12:02:06.077 INFO:tasks.workunit.client.0.smithi032.stdout: "ready_epoch": 0, 2024-09-13T12:02:06.077 INFO:tasks.workunit.client.0.smithi032.stdout: "last_epoch_started": 0, 2024-09-13T12:02:06.077 INFO:tasks.workunit.client.0.smithi032.stdout: "last_epoch_clean": 0, 2024-09-13T12:02:06.077 INFO:tasks.workunit.client.0.smithi032.stdout: "source_version": "0'0", 2024-09-13T12:02:06.077 INFO:tasks.workunit.client.0.smithi032.stdout: "target_version": "0'0" 2024-09-13T12:02:06.077 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.077 INFO:tasks.workunit.client.0.smithi032.stdout: "last_change": "28", 2024-09-13T12:02:06.077 INFO:tasks.workunit.client.0.smithi032.stdout: "last_force_op_resend": "0", 2024-09-13T12:02:06.077 INFO:tasks.workunit.client.0.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T12:02:06.077 INFO:tasks.workunit.client.0.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T12:02:06.077 INFO:tasks.workunit.client.0.smithi032.stdout: "auid": 0, 2024-09-13T12:02:06.077 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_mode": "selfmanaged", 2024-09-13T12:02:06.077 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_seq": 0, 2024-09-13T12:02:06.078 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_epoch": 0, 2024-09-13T12:02:06.078 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_snaps": [], 2024-09-13T12:02:06.078 INFO:tasks.workunit.client.0.smithi032.stdout: "removed_snaps": "[]", 2024-09-13T12:02:06.078 INFO:tasks.workunit.client.0.smithi032.stdout: "quota_max_bytes": 0, 2024-09-13T12:02:06.078 INFO:tasks.workunit.client.0.smithi032.stdout: "quota_max_objects": 0, 2024-09-13T12:02:06.078 INFO:tasks.workunit.client.0.smithi032.stdout: "tiers": [], 2024-09-13T12:02:06.078 INFO:tasks.workunit.client.0.smithi032.stdout: "tier_of": -1, 2024-09-13T12:02:06.078 INFO:tasks.workunit.client.0.smithi032.stdout: "read_tier": -1, 2024-09-13T12:02:06.078 INFO:tasks.workunit.client.0.smithi032.stdout: "write_tier": -1, 2024-09-13T12:02:06.078 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_mode": "none", 2024-09-13T12:02:06.078 INFO:tasks.workunit.client.0.smithi032.stdout: "target_max_bytes": 0, 2024-09-13T12:02:06.078 INFO:tasks.workunit.client.0.smithi032.stdout: "target_max_objects": 0, 2024-09-13T12:02:06.078 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T12:02:06.079 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T12:02:06.079 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T12:02:06.079 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-13T12:02:06.079 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-13T12:02:06.079 INFO:tasks.workunit.client.0.smithi032.stdout: "erasure_code_profile": "", 2024-09-13T12:02:06.079 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_params": { 2024-09-13T12:02:06.079 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "none" 2024-09-13T12:02:06.079 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.079 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_period": 0, 2024-09-13T12:02:06.079 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_count": 0, 2024-09-13T12:02:06.079 INFO:tasks.workunit.client.0.smithi032.stdout: "use_gmt_hitset": true, 2024-09-13T12:02:06.079 INFO:tasks.workunit.client.0.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-13T12:02:06.079 INFO:tasks.workunit.client.0.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-13T12:02:06.079 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T12:02:06.079 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-13T12:02:06.080 INFO:tasks.workunit.client.0.smithi032.stdout: "grade_table": [], 2024-09-13T12:02:06.080 INFO:tasks.workunit.client.0.smithi032.stdout: "stripe_width": 0, 2024-09-13T12:02:06.080 INFO:tasks.workunit.client.0.smithi032.stdout: "expected_num_objects": 0, 2024-09-13T12:02:06.080 INFO:tasks.workunit.client.0.smithi032.stdout: "fast_read": false, 2024-09-13T12:02:06.080 INFO:tasks.workunit.client.0.smithi032.stdout: "options": {}, 2024-09-13T12:02:06.080 INFO:tasks.workunit.client.0.smithi032.stdout: "application_metadata": {}, 2024-09-13T12:02:06.080 INFO:tasks.workunit.client.0.smithi032.stdout: "read_balance": { 2024-09-13T12:02:06.080 INFO:tasks.workunit.client.0.smithi032.stdout: "score_type": "Fair distribution", 2024-09-13T12:02:06.080 INFO:tasks.workunit.client.0.smithi032.stdout: "score_acting": 2.9900000095367432, 2024-09-13T12:02:06.080 INFO:tasks.workunit.client.0.smithi032.stdout: "score_stable": 2.9900000095367432, 2024-09-13T12:02:06.080 INFO:tasks.workunit.client.0.smithi032.stdout: "optimal_score": 0.67000001668930054, 2024-09-13T12:02:06.080 INFO:tasks.workunit.client.0.smithi032.stdout: "raw_score_acting": 2, 2024-09-13T12:02:06.080 INFO:tasks.workunit.client.0.smithi032.stdout: "raw_score_stable": 2, 2024-09-13T12:02:06.080 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_affinity_weighted": 1, 2024-09-13T12:02:06.080 INFO:tasks.workunit.client.0.smithi032.stdout: "average_primary_affinity": 1, 2024-09-13T12:02:06.081 INFO:tasks.workunit.client.0.smithi032.stdout: "average_primary_affinity_weighted": 1 2024-09-13T12:02:06.081 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.081 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.081 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:02:06.081 INFO:tasks.workunit.client.0.smithi032.stdout: "osds": [ 2024-09-13T12:02:06.081 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.081 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 0, 2024-09-13T12:02:06.081 INFO:tasks.workunit.client.0.smithi032.stdout: "uuid": "42e90f08-426a-4b5b-ad78-7bd0b8144f62", 2024-09-13T12:02:06.081 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 1, 2024-09-13T12:02:06.081 INFO:tasks.workunit.client.0.smithi032.stdout: "in": 1, 2024-09-13T12:02:06.081 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 1, 2024-09-13T12:02:06.081 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_affinity": 1, 2024-09-13T12:02:06.081 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_begin": 5, 2024-09-13T12:02:06.082 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_end": 28, 2024-09-13T12:02:06.082 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 35, 2024-09-13T12:02:06.082 INFO:tasks.workunit.client.0.smithi032.stdout: "up_thru": 39, 2024-09-13T12:02:06.082 INFO:tasks.workunit.client.0.smithi032.stdout: "down_at": 29, 2024-09-13T12:02:06.082 INFO:tasks.workunit.client.0.smithi032.stdout: "lost_at": 0, 2024-09-13T12:02:06.082 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addrs": { 2024-09-13T12:02:06.082 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:02:06.082 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.082 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:02:06.082 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6802", 2024-09-13T12:02:06.082 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 442043230 2024-09-13T12:02:06.082 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.082 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.082 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:02:06.082 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6803", 2024-09-13T12:02:06.083 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 442043230 2024-09-13T12:02:06.083 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.083 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:02:06.083 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.083 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addrs": { 2024-09-13T12:02:06.083 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:02:06.083 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.083 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:02:06.083 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6804", 2024-09-13T12:02:06.083 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 442043230 2024-09-13T12:02:06.083 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.083 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.083 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:02:06.084 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6805", 2024-09-13T12:02:06.084 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 442043230 2024-09-13T12:02:06.084 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.084 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:02:06.084 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.084 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-13T12:02:06.084 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:02:06.084 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.084 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:02:06.084 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6808", 2024-09-13T12:02:06.084 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 442043230 2024-09-13T12:02:06.084 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.084 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.084 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:02:06.085 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6809", 2024-09-13T12:02:06.085 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 442043230 2024-09-13T12:02:06.085 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.085 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:02:06.085 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.085 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-13T12:02:06.085 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:02:06.085 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.085 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:02:06.085 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6806", 2024-09-13T12:02:06.085 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 442043230 2024-09-13T12:02:06.085 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.085 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.085 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:02:06.086 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6807", 2024-09-13T12:02:06.086 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 442043230 2024-09-13T12:02:06.086 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.086 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:02:06.086 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.086 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addr": "127.0.0.1:6803/442043230", 2024-09-13T12:02:06.086 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addr": "127.0.0.1:6805/442043230", 2024-09-13T12:02:06.086 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addr": "127.0.0.1:6809/442043230", 2024-09-13T12:02:06.086 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addr": "127.0.0.1:6807/442043230", 2024-09-13T12:02:06.086 INFO:tasks.workunit.client.0.smithi032.stdout: "state": [ 2024-09-13T12:02:06.086 INFO:tasks.workunit.client.0.smithi032.stdout: "exists", 2024-09-13T12:02:06.086 INFO:tasks.workunit.client.0.smithi032.stdout: "up" 2024-09-13T12:02:06.087 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:02:06.087 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.087 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.087 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 1, 2024-09-13T12:02:06.087 INFO:tasks.workunit.client.0.smithi032.stdout: "uuid": "45eb455b-ec14-492a-a47e-4e0a84ac4a8c", 2024-09-13T12:02:06.087 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 0, 2024-09-13T12:02:06.087 INFO:tasks.workunit.client.0.smithi032.stdout: "in": 1, 2024-09-13T12:02:06.087 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 1, 2024-09-13T12:02:06.087 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_affinity": 1, 2024-09-13T12:02:06.087 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_begin": 0, 2024-09-13T12:02:06.087 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_end": 0, 2024-09-13T12:02:06.087 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 10, 2024-09-13T12:02:06.087 INFO:tasks.workunit.client.0.smithi032.stdout: "up_thru": 23, 2024-09-13T12:02:06.087 INFO:tasks.workunit.client.0.smithi032.stdout: "down_at": 30, 2024-09-13T12:02:06.087 INFO:tasks.workunit.client.0.smithi032.stdout: "lost_at": 0, 2024-09-13T12:02:06.088 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addrs": { 2024-09-13T12:02:06.088 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:02:06.088 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.088 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:02:06.088 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6810", 2024-09-13T12:02:06.088 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 671786448 2024-09-13T12:02:06.088 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.088 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.088 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:02:06.088 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6811", 2024-09-13T12:02:06.088 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 671786448 2024-09-13T12:02:06.088 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.088 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:02:06.088 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.089 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addrs": { 2024-09-13T12:02:06.089 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:02:06.089 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.089 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:02:06.089 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6812", 2024-09-13T12:02:06.089 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 671786448 2024-09-13T12:02:06.089 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.089 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.089 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:02:06.089 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6813", 2024-09-13T12:02:06.089 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 671786448 2024-09-13T12:02:06.089 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.089 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:02:06.089 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.090 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-13T12:02:06.090 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:02:06.090 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.090 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:02:06.090 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6816", 2024-09-13T12:02:06.090 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 671786448 2024-09-13T12:02:06.090 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.090 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.090 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:02:06.090 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6817", 2024-09-13T12:02:06.090 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 671786448 2024-09-13T12:02:06.090 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.090 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:02:06.090 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.091 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-13T12:02:06.091 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:02:06.091 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.091 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:02:06.091 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6814", 2024-09-13T12:02:06.091 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 671786448 2024-09-13T12:02:06.091 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.091 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.091 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:02:06.091 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6815", 2024-09-13T12:02:06.091 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 671786448 2024-09-13T12:02:06.091 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.091 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:02:06.091 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.091 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addr": "127.0.0.1:6811/671786448", 2024-09-13T12:02:06.092 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addr": "127.0.0.1:6813/671786448", 2024-09-13T12:02:06.092 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addr": "127.0.0.1:6817/671786448", 2024-09-13T12:02:06.092 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addr": "127.0.0.1:6815/671786448", 2024-09-13T12:02:06.092 INFO:tasks.workunit.client.0.smithi032.stdout: "state": [ 2024-09-13T12:02:06.092 INFO:tasks.workunit.client.0.smithi032.stdout: "exists" 2024-09-13T12:02:06.092 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:02:06.092 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.092 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.092 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2, 2024-09-13T12:02:06.092 INFO:tasks.workunit.client.0.smithi032.stdout: "uuid": "4607d722-ea41-4ddb-a0cd-5570afca8326", 2024-09-13T12:02:06.092 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 1, 2024-09-13T12:02:06.092 INFO:tasks.workunit.client.0.smithi032.stdout: "in": 1, 2024-09-13T12:02:06.092 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 1, 2024-09-13T12:02:06.092 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_affinity": 1, 2024-09-13T12:02:06.092 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_begin": 15, 2024-09-13T12:02:06.093 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_end": 28, 2024-09-13T12:02:06.093 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 37, 2024-09-13T12:02:06.093 INFO:tasks.workunit.client.0.smithi032.stdout: "up_thru": 0, 2024-09-13T12:02:06.093 INFO:tasks.workunit.client.0.smithi032.stdout: "down_at": 31, 2024-09-13T12:02:06.093 INFO:tasks.workunit.client.0.smithi032.stdout: "lost_at": 0, 2024-09-13T12:02:06.093 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addrs": { 2024-09-13T12:02:06.093 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:02:06.093 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.093 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:02:06.093 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6810", 2024-09-13T12:02:06.093 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3448235176 2024-09-13T12:02:06.093 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.093 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.094 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:02:06.094 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6811", 2024-09-13T12:02:06.094 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3448235176 2024-09-13T12:02:06.094 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.094 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:02:06.094 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.094 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addrs": { 2024-09-13T12:02:06.094 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:02:06.094 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.094 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:02:06.094 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6812", 2024-09-13T12:02:06.094 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3448235176 2024-09-13T12:02:06.094 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.094 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.094 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:02:06.095 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6813", 2024-09-13T12:02:06.095 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3448235176 2024-09-13T12:02:06.095 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.095 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:02:06.095 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.095 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-13T12:02:06.095 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:02:06.095 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.095 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:02:06.095 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6816", 2024-09-13T12:02:06.095 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3448235176 2024-09-13T12:02:06.095 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.095 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.095 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:02:06.095 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6817", 2024-09-13T12:02:06.095 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3448235176 2024-09-13T12:02:06.096 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.096 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:02:06.096 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.096 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-13T12:02:06.096 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:02:06.096 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.096 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:02:06.096 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6814", 2024-09-13T12:02:06.096 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3448235176 2024-09-13T12:02:06.096 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.096 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.096 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:02:06.096 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6815", 2024-09-13T12:02:06.096 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3448235176 2024-09-13T12:02:06.096 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.097 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:02:06.097 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.097 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addr": "127.0.0.1:6811/3448235176", 2024-09-13T12:02:06.097 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addr": "127.0.0.1:6813/3448235176", 2024-09-13T12:02:06.097 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3448235176", 2024-09-13T12:02:06.097 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3448235176", 2024-09-13T12:02:06.097 INFO:tasks.workunit.client.0.smithi032.stdout: "state": [ 2024-09-13T12:02:06.097 INFO:tasks.workunit.client.0.smithi032.stdout: "exists", 2024-09-13T12:02:06.097 INFO:tasks.workunit.client.0.smithi032.stdout: "up" 2024-09-13T12:02:06.097 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:02:06.097 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.097 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:02:06.097 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_xinfo": [ 2024-09-13T12:02:06.097 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.098 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 0, 2024-09-13T12:02:06.098 INFO:tasks.workunit.client.0.smithi032.stdout: "down_stamp": "2024-09-13T12:01:13.787177+0000", 2024-09-13T12:02:06.098 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_probability": 0, 2024-09-13T12:02:06.098 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_interval": 0, 2024-09-13T12:02:06.098 INFO:tasks.workunit.client.0.smithi032.stdout: "features": 4540719139924082687, 2024-09-13T12:02:06.098 INFO:tasks.workunit.client.0.smithi032.stdout: "old_weight": 0, 2024-09-13T12:02:06.098 INFO:tasks.workunit.client.0.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-13T11:59:14.377494+0000", 2024-09-13T12:02:06.098 INFO:tasks.workunit.client.0.smithi032.stdout: "dead_epoch": 0 2024-09-13T12:02:06.098 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.098 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.098 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 1, 2024-09-13T12:02:06.098 INFO:tasks.workunit.client.0.smithi032.stdout: "down_stamp": "2024-09-13T12:01:14.340651+0000", 2024-09-13T12:02:06.098 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_probability": 0, 2024-09-13T12:02:06.098 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_interval": 0, 2024-09-13T12:02:06.099 INFO:tasks.workunit.client.0.smithi032.stdout: "features": 4540719139924082687, 2024-09-13T12:02:06.099 INFO:tasks.workunit.client.0.smithi032.stdout: "old_weight": 0, 2024-09-13T12:02:06.099 INFO:tasks.workunit.client.0.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-13T11:59:48.053596+0000", 2024-09-13T12:02:06.099 INFO:tasks.workunit.client.0.smithi032.stdout: "dead_epoch": 0 2024-09-13T12:02:06.099 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.099 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.099 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2, 2024-09-13T12:02:06.099 INFO:tasks.workunit.client.0.smithi032.stdout: "down_stamp": "2024-09-13T12:01:14.931359+0000", 2024-09-13T12:02:06.099 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_probability": 0, 2024-09-13T12:02:06.099 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_interval": 0, 2024-09-13T12:02:06.099 INFO:tasks.workunit.client.0.smithi032.stdout: "features": 4540719139924082687, 2024-09-13T12:02:06.099 INFO:tasks.workunit.client.0.smithi032.stdout: "old_weight": 0, 2024-09-13T12:02:06.099 INFO:tasks.workunit.client.0.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-13T12:00:23.382989+0000", 2024-09-13T12:02:06.099 INFO:tasks.workunit.client.0.smithi032.stdout: "dead_epoch": 0 2024-09-13T12:02:06.099 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.100 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:02:06.100 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_upmap": [], 2024-09-13T12:02:06.100 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_upmap_items": [], 2024-09-13T12:02:06.100 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_upmap_primaries": [], 2024-09-13T12:02:06.100 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_temp": [], 2024-09-13T12:02:06.100 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_temp": [], 2024-09-13T12:02:06.100 INFO:tasks.workunit.client.0.smithi032.stdout: "blocklist": {}, 2024-09-13T12:02:06.100 INFO:tasks.workunit.client.0.smithi032.stdout: "range_blocklist": {}, 2024-09-13T12:02:06.100 INFO:tasks.workunit.client.0.smithi032.stdout: "erasure_code_profiles": { 2024-09-13T12:02:06.100 INFO:tasks.workunit.client.0.smithi032.stdout: "default": { 2024-09-13T12:02:06.100 INFO:tasks.workunit.client.0.smithi032.stdout: "k": "2", 2024-09-13T12:02:06.100 INFO:tasks.workunit.client.0.smithi032.stdout: "m": "2", 2024-09-13T12:02:06.100 INFO:tasks.workunit.client.0.smithi032.stdout: "plugin": "jerasure", 2024-09-13T12:02:06.100 INFO:tasks.workunit.client.0.smithi032.stdout: "technique": "reed_sol_van" 2024-09-13T12:02:06.100 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.101 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.101 INFO:tasks.workunit.client.0.smithi032.stdout: "removed_snaps_queue": [], 2024-09-13T12:02:06.101 INFO:tasks.workunit.client.0.smithi032.stdout: "new_removed_snaps": [], 2024-09-13T12:02:06.101 INFO:tasks.workunit.client.0.smithi032.stdout: "new_purged_snaps": [], 2024-09-13T12:02:06.101 INFO:tasks.workunit.client.0.smithi032.stdout: "crush_node_flags": {}, 2024-09-13T12:02:06.101 INFO:tasks.workunit.client.0.smithi032.stdout: "device_class_flags": {}, 2024-09-13T12:02:06.101 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_mode": { 2024-09-13T12:02:06.101 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_mode_enabled": false, 2024-09-13T12:02:06.101 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_bucket_count": 0, 2024-09-13T12:02:06.101 INFO:tasks.workunit.client.0.smithi032.stdout: "degraded_stretch_mode": 0, 2024-09-13T12:02:06.101 INFO:tasks.workunit.client.0.smithi032.stdout: "recovering_stretch_mode": 0, 2024-09-13T12:02:06.101 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_mode_bucket": 0 2024-09-13T12:02:06.101 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.101 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.102 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_metadata": [ 2024-09-13T12:02:06.102 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.102 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:02:06.102 INFO:tasks.workunit.client.0.smithi032.stdout: "arch": "x86_64", 2024-09-13T12:02:06.102 INFO:tasks.workunit.client.0.smithi032.stdout: "back_addr": "[v2:127.0.0.1:6804/442043230,v1:127.0.0.1:6805/442043230]", 2024-09-13T12:02:06.102 INFO:tasks.workunit.client.0.smithi032.stdout: "back_iface": "", 2024-09-13T12:02:06.102 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs": "1", 2024-09-13T12:02:06.102 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_db": "0", 2024-09-13T12:02:06.102 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_wal": "0", 2024-09-13T12:02:06.102 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_single_shared_device": "1", 2024-09-13T12:02:06.102 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_allocation_from_file": "0", 2024-09-13T12:02:06.102 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_access_mode": "file", 2024-09-13T12:02:06.102 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_block_size": "4096", 2024-09-13T12:02:06.102 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_devices": "sda", 2024-09-13T12:02:06.102 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-13T12:02:06.102 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-13T12:02:06.103 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-13T12:02:06.103 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_rotational": "1", 2024-09-13T12:02:06.103 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_size": "107374182400", 2024-09-13T12:02:06.103 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_support_discard": "1", 2024-09-13T12:02:06.103 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_type": "hdd", 2024-09-13T12:02:06.103 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_min_alloc_size": "4096", 2024-09-13T12:02:06.103 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_release": "squid", 2024-09-13T12:02:06.103 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:02:06.103 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_short": "19.3.0-4697-gd58f7306", 2024-09-13T12:02:06.103 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_when_created": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:02:06.103 INFO:tasks.workunit.client.0.smithi032.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-13T12:02:06.103 INFO:tasks.workunit.client.0.smithi032.stdout: "created_at": "2024-09-13T11:58:47.576290Z", 2024-09-13T12:02:06.103 INFO:tasks.workunit.client.0.smithi032.stdout: "default_device_class": "hdd", 2024-09-13T12:02:06.103 INFO:tasks.workunit.client.0.smithi032.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4J2LL", 2024-09-13T12:02:06.103 INFO:tasks.workunit.client.0.smithi032.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-13T12:02:06.103 INFO:tasks.workunit.client.0.smithi032.stdout: "devices": "sda", 2024-09-13T12:02:06.104 INFO:tasks.workunit.client.0.smithi032.stdout: "distro": "centos", 2024-09-13T12:02:06.104 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_description": "CentOS Stream 9", 2024-09-13T12:02:06.104 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_version": "9", 2024-09-13T12:02:06.104 INFO:tasks.workunit.client.0.smithi032.stdout: "front_addr": "[v2:127.0.0.1:6802/442043230,v1:127.0.0.1:6803/442043230]", 2024-09-13T12:02:06.104 INFO:tasks.workunit.client.0.smithi032.stdout: "front_iface": "", 2024-09-13T12:02:06.104 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/442043230,v1:127.0.0.1:6809/442043230]", 2024-09-13T12:02:06.104 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/442043230,v1:127.0.0.1:6807/442043230]", 2024-09-13T12:02:06.104 INFO:tasks.workunit.client.0.smithi032.stdout: "hostname": "smithi032", 2024-09-13T12:02:06.104 INFO:tasks.workunit.client.0.smithi032.stdout: "journal_rotational": "1", 2024-09-13T12:02:06.104 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-13T12:02:06.104 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-13T12:02:06.104 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_swap_kb": "0", 2024-09-13T12:02:06.104 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_total_kb": "32493484", 2024-09-13T12:02:06.104 INFO:tasks.workunit.client.0.smithi032.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-13T12:02:06.104 INFO:tasks.workunit.client.0.smithi032.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-13T12:02:06.105 INFO:tasks.workunit.client.0.smithi032.stdout: "os": "Linux", 2024-09-13T12:02:06.105 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_data": "td/divergent-priors/0", 2024-09-13T12:02:06.105 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_objectstore": "bluestore", 2024-09-13T12:02:06.105 INFO:tasks.workunit.client.0.smithi032.stdout: "osdspec_affinity": "", 2024-09-13T12:02:06.105 INFO:tasks.workunit.client.0.smithi032.stdout: "rotational": "1" 2024-09-13T12:02:06.105 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.105 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.105 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 1, 2024-09-13T12:02:06.105 INFO:tasks.workunit.client.0.smithi032.stdout: "arch": "x86_64", 2024-09-13T12:02:06.105 INFO:tasks.workunit.client.0.smithi032.stdout: "back_addr": "[v2:127.0.0.1:6812/671786448,v1:127.0.0.1:6813/671786448]", 2024-09-13T12:02:06.105 INFO:tasks.workunit.client.0.smithi032.stdout: "back_iface": "", 2024-09-13T12:02:06.105 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs": "1", 2024-09-13T12:02:06.105 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_db": "0", 2024-09-13T12:02:06.105 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_wal": "0", 2024-09-13T12:02:06.105 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_single_shared_device": "1", 2024-09-13T12:02:06.106 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_allocation_from_file": "0", 2024-09-13T12:02:06.106 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_access_mode": "file", 2024-09-13T12:02:06.106 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_block_size": "4096", 2024-09-13T12:02:06.106 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_devices": "sda", 2024-09-13T12:02:06.106 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-13T12:02:06.106 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-13T12:02:06.106 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-13T12:02:06.106 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_rotational": "1", 2024-09-13T12:02:06.106 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_size": "107374182400", 2024-09-13T12:02:06.106 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_support_discard": "1", 2024-09-13T12:02:06.106 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_type": "hdd", 2024-09-13T12:02:06.106 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_min_alloc_size": "4096", 2024-09-13T12:02:06.106 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_release": "squid", 2024-09-13T12:02:06.106 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:02:06.106 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_short": "19.3.0-4697-gd58f7306", 2024-09-13T12:02:06.106 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_when_created": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:02:06.107 INFO:tasks.workunit.client.0.smithi032.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-13T12:02:06.107 INFO:tasks.workunit.client.0.smithi032.stdout: "created_at": "2024-09-13T11:59:21.313330Z", 2024-09-13T12:02:06.107 INFO:tasks.workunit.client.0.smithi032.stdout: "default_device_class": "hdd", 2024-09-13T12:02:06.107 INFO:tasks.workunit.client.0.smithi032.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4J2LL", 2024-09-13T12:02:06.107 INFO:tasks.workunit.client.0.smithi032.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-13T12:02:06.107 INFO:tasks.workunit.client.0.smithi032.stdout: "devices": "sda", 2024-09-13T12:02:06.107 INFO:tasks.workunit.client.0.smithi032.stdout: "distro": "centos", 2024-09-13T12:02:06.107 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_description": "CentOS Stream 9", 2024-09-13T12:02:06.107 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_version": "9", 2024-09-13T12:02:06.107 INFO:tasks.workunit.client.0.smithi032.stdout: "front_addr": "[v2:127.0.0.1:6810/671786448,v1:127.0.0.1:6811/671786448]", 2024-09-13T12:02:06.107 INFO:tasks.workunit.client.0.smithi032.stdout: "front_iface": "", 2024-09-13T12:02:06.107 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/671786448,v1:127.0.0.1:6817/671786448]", 2024-09-13T12:02:06.107 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/671786448,v1:127.0.0.1:6815/671786448]", 2024-09-13T12:02:06.107 INFO:tasks.workunit.client.0.smithi032.stdout: "hostname": "smithi032", 2024-09-13T12:02:06.107 INFO:tasks.workunit.client.0.smithi032.stdout: "journal_rotational": "1", 2024-09-13T12:02:06.107 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-13T12:02:06.108 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-13T12:02:06.108 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_swap_kb": "0", 2024-09-13T12:02:06.108 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_total_kb": "32493484", 2024-09-13T12:02:06.108 INFO:tasks.workunit.client.0.smithi032.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-13T12:02:06.108 INFO:tasks.workunit.client.0.smithi032.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-13T12:02:06.108 INFO:tasks.workunit.client.0.smithi032.stdout: "os": "Linux", 2024-09-13T12:02:06.108 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_data": "td/divergent-priors/1", 2024-09-13T12:02:06.108 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_objectstore": "bluestore", 2024-09-13T12:02:06.108 INFO:tasks.workunit.client.0.smithi032.stdout: "osdspec_affinity": "", 2024-09-13T12:02:06.108 INFO:tasks.workunit.client.0.smithi032.stdout: "rotational": "1" 2024-09-13T12:02:06.108 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.108 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.108 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:02:06.108 INFO:tasks.workunit.client.0.smithi032.stdout: "arch": "x86_64", 2024-09-13T12:02:06.108 INFO:tasks.workunit.client.0.smithi032.stdout: "back_addr": "[v2:127.0.0.1:6812/3448235176,v1:127.0.0.1:6813/3448235176]", 2024-09-13T12:02:06.108 INFO:tasks.workunit.client.0.smithi032.stdout: "back_iface": "", 2024-09-13T12:02:06.109 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs": "1", 2024-09-13T12:02:06.109 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_db": "0", 2024-09-13T12:02:06.109 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_wal": "0", 2024-09-13T12:02:06.109 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_single_shared_device": "1", 2024-09-13T12:02:06.109 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_allocation_from_file": "0", 2024-09-13T12:02:06.109 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_access_mode": "file", 2024-09-13T12:02:06.109 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_block_size": "4096", 2024-09-13T12:02:06.109 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_devices": "sda", 2024-09-13T12:02:06.109 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-13T12:02:06.109 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-13T12:02:06.109 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-13T12:02:06.109 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_rotational": "1", 2024-09-13T12:02:06.109 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_size": "107374182400", 2024-09-13T12:02:06.109 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_support_discard": "1", 2024-09-13T12:02:06.109 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_type": "hdd", 2024-09-13T12:02:06.109 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_min_alloc_size": "4096", 2024-09-13T12:02:06.110 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_release": "squid", 2024-09-13T12:02:06.110 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:02:06.110 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_short": "19.3.0-4697-gd58f7306", 2024-09-13T12:02:06.110 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_when_created": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:02:06.110 INFO:tasks.workunit.client.0.smithi032.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-13T12:02:06.110 INFO:tasks.workunit.client.0.smithi032.stdout: "created_at": "2024-09-13T11:59:55.139843Z", 2024-09-13T12:02:06.110 INFO:tasks.workunit.client.0.smithi032.stdout: "default_device_class": "hdd", 2024-09-13T12:02:06.110 INFO:tasks.workunit.client.0.smithi032.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4J2LL", 2024-09-13T12:02:06.110 INFO:tasks.workunit.client.0.smithi032.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-13T12:02:06.110 INFO:tasks.workunit.client.0.smithi032.stdout: "devices": "sda", 2024-09-13T12:02:06.110 INFO:tasks.workunit.client.0.smithi032.stdout: "distro": "centos", 2024-09-13T12:02:06.110 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_description": "CentOS Stream 9", 2024-09-13T12:02:06.110 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_version": "9", 2024-09-13T12:02:06.110 INFO:tasks.workunit.client.0.smithi032.stdout: "front_addr": "[v2:127.0.0.1:6810/3448235176,v1:127.0.0.1:6811/3448235176]", 2024-09-13T12:02:06.111 INFO:tasks.workunit.client.0.smithi032.stdout: "front_iface": "", 2024-09-13T12:02:06.111 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3448235176,v1:127.0.0.1:6817/3448235176]", 2024-09-13T12:02:06.111 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3448235176,v1:127.0.0.1:6815/3448235176]", 2024-09-13T12:02:06.111 INFO:tasks.workunit.client.0.smithi032.stdout: "hostname": "smithi032", 2024-09-13T12:02:06.111 INFO:tasks.workunit.client.0.smithi032.stdout: "journal_rotational": "1", 2024-09-13T12:02:06.111 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-13T12:02:06.111 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-13T12:02:06.111 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_swap_kb": "0", 2024-09-13T12:02:06.111 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_total_kb": "32493484", 2024-09-13T12:02:06.111 INFO:tasks.workunit.client.0.smithi032.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-13T12:02:06.111 INFO:tasks.workunit.client.0.smithi032.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-13T12:02:06.111 INFO:tasks.workunit.client.0.smithi032.stdout: "os": "Linux", 2024-09-13T12:02:06.111 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_data": "td/divergent-priors/2", 2024-09-13T12:02:06.111 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_objectstore": "bluestore", 2024-09-13T12:02:06.112 INFO:tasks.workunit.client.0.smithi032.stdout: "osdspec_affinity": "", 2024-09-13T12:02:06.112 INFO:tasks.workunit.client.0.smithi032.stdout: "rotational": "1" 2024-09-13T12:02:06.112 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.112 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:02:06.112 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap_clean_epochs": { 2024-09-13T12:02:06.112 INFO:tasks.workunit.client.0.smithi032.stdout: "min_last_epoch_clean": 0, 2024-09-13T12:02:06.112 INFO:tasks.workunit.client.0.smithi032.stdout: "last_epoch_clean": { 2024-09-13T12:02:06.112 INFO:tasks.workunit.client.0.smithi032.stdout: "per_pool": [] 2024-09-13T12:02:06.112 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.112 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_epochs": [ 2024-09-13T12:02:06.112 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.112 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:02:06.112 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 35 2024-09-13T12:02:06.112 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.112 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.113 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:02:06.113 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 37 2024-09-13T12:02:06.113 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.113 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:02:06.113 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.113 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap_first_committed": 1, 2024-09-13T12:02:06.113 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap_last_committed": 40, 2024-09-13T12:02:06.113 INFO:tasks.workunit.client.0.smithi032.stdout: "crushmap": { 2024-09-13T12:02:06.113 INFO:tasks.workunit.client.0.smithi032.stdout: "devices": [ 2024-09-13T12:02:06.113 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.113 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:02:06.113 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "osd.0", 2024-09-13T12:02:06.113 INFO:tasks.workunit.client.0.smithi032.stdout: "class": "hdd" 2024-09-13T12:02:06.113 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.113 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.114 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 1, 2024-09-13T12:02:06.114 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "osd.1", 2024-09-13T12:02:06.114 INFO:tasks.workunit.client.0.smithi032.stdout: "class": "hdd" 2024-09-13T12:02:06.114 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.114 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.114 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:02:06.114 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "osd.2", 2024-09-13T12:02:06.114 INFO:tasks.workunit.client.0.smithi032.stdout: "class": "hdd" 2024-09-13T12:02:06.114 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.114 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:02:06.114 INFO:tasks.workunit.client.0.smithi032.stdout: "types": [ 2024-09-13T12:02:06.114 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.114 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 0, 2024-09-13T12:02:06.114 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "osd" 2024-09-13T12:02:06.114 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.114 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.115 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 1, 2024-09-13T12:02:06.115 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "host" 2024-09-13T12:02:06.115 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.115 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.115 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 2, 2024-09-13T12:02:06.115 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "chassis" 2024-09-13T12:02:06.115 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.115 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.115 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 3, 2024-09-13T12:02:06.115 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "rack" 2024-09-13T12:02:06.115 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.115 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.115 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 4, 2024-09-13T12:02:06.115 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "row" 2024-09-13T12:02:06.115 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.115 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.116 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 5, 2024-09-13T12:02:06.116 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "pdu" 2024-09-13T12:02:06.116 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.116 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.116 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 6, 2024-09-13T12:02:06.116 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "pod" 2024-09-13T12:02:06.116 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.116 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.116 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 7, 2024-09-13T12:02:06.116 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "room" 2024-09-13T12:02:06.116 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.116 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.116 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 8, 2024-09-13T12:02:06.116 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "datacenter" 2024-09-13T12:02:06.116 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.116 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.117 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 9, 2024-09-13T12:02:06.117 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "zone" 2024-09-13T12:02:06.117 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.117 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.117 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 10, 2024-09-13T12:02:06.117 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "region" 2024-09-13T12:02:06.117 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.117 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.117 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 11, 2024-09-13T12:02:06.117 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "root" 2024-09-13T12:02:06.117 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.117 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:02:06.117 INFO:tasks.workunit.client.0.smithi032.stdout: "buckets": [ 2024-09-13T12:02:06.117 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.117 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -1, 2024-09-13T12:02:06.118 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "default", 2024-09-13T12:02:06.118 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 11, 2024-09-13T12:02:06.118 INFO:tasks.workunit.client.0.smithi032.stdout: "type_name": "root", 2024-09-13T12:02:06.118 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:02:06.118 INFO:tasks.workunit.client.0.smithi032.stdout: "alg": "straw2", 2024-09-13T12:02:06.118 INFO:tasks.workunit.client.0.smithi032.stdout: "hash": "rjenkins1", 2024-09-13T12:02:06.118 INFO:tasks.workunit.client.0.smithi032.stdout: "items": [ 2024-09-13T12:02:06.118 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.118 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -3, 2024-09-13T12:02:06.118 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:02:06.118 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 0 2024-09-13T12:02:06.118 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.118 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:02:06.118 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.118 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.119 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -2, 2024-09-13T12:02:06.119 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "default~hdd", 2024-09-13T12:02:06.119 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 11, 2024-09-13T12:02:06.119 INFO:tasks.workunit.client.0.smithi032.stdout: "type_name": "root", 2024-09-13T12:02:06.119 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:02:06.119 INFO:tasks.workunit.client.0.smithi032.stdout: "alg": "straw2", 2024-09-13T12:02:06.119 INFO:tasks.workunit.client.0.smithi032.stdout: "hash": "rjenkins1", 2024-09-13T12:02:06.119 INFO:tasks.workunit.client.0.smithi032.stdout: "items": [ 2024-09-13T12:02:06.119 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.119 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -4, 2024-09-13T12:02:06.119 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:02:06.119 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 0 2024-09-13T12:02:06.119 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.119 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:02:06.119 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.119 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.119 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -3, 2024-09-13T12:02:06.120 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "smithi032", 2024-09-13T12:02:06.120 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 1, 2024-09-13T12:02:06.120 INFO:tasks.workunit.client.0.smithi032.stdout: "type_name": "host", 2024-09-13T12:02:06.120 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:02:06.120 INFO:tasks.workunit.client.0.smithi032.stdout: "alg": "straw2", 2024-09-13T12:02:06.120 INFO:tasks.workunit.client.0.smithi032.stdout: "hash": "rjenkins1", 2024-09-13T12:02:06.120 INFO:tasks.workunit.client.0.smithi032.stdout: "items": [ 2024-09-13T12:02:06.120 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.120 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:02:06.120 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:02:06.120 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 0 2024-09-13T12:02:06.120 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.120 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.120 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 1, 2024-09-13T12:02:06.120 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:02:06.121 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 1 2024-09-13T12:02:06.121 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.121 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.121 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:02:06.121 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:02:06.121 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 2 2024-09-13T12:02:06.121 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.121 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:02:06.121 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.121 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.121 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -4, 2024-09-13T12:02:06.121 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "smithi032~hdd", 2024-09-13T12:02:06.121 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 1, 2024-09-13T12:02:06.122 INFO:tasks.workunit.client.0.smithi032.stdout: "type_name": "host", 2024-09-13T12:02:06.122 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:02:06.122 INFO:tasks.workunit.client.0.smithi032.stdout: "alg": "straw2", 2024-09-13T12:02:06.122 INFO:tasks.workunit.client.0.smithi032.stdout: "hash": "rjenkins1", 2024-09-13T12:02:06.122 INFO:tasks.workunit.client.0.smithi032.stdout: "items": [ 2024-09-13T12:02:06.122 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.122 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:02:06.122 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:02:06.122 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 0 2024-09-13T12:02:06.122 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.122 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.122 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 1, 2024-09-13T12:02:06.122 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:02:06.122 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 1 2024-09-13T12:02:06.123 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.123 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.123 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:02:06.123 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:02:06.123 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 2 2024-09-13T12:02:06.123 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.123 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:02:06.123 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.123 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:02:06.123 INFO:tasks.workunit.client.0.smithi032.stdout: "rules": [ 2024-09-13T12:02:06.123 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.123 INFO:tasks.workunit.client.0.smithi032.stdout: "rule_id": 0, 2024-09-13T12:02:06.123 INFO:tasks.workunit.client.0.smithi032.stdout: "rule_name": "replicated_rule", 2024-09-13T12:02:06.123 INFO:tasks.workunit.client.0.smithi032.stdout: "type": 1, 2024-09-13T12:02:06.123 INFO:tasks.workunit.client.0.smithi032.stdout: "steps": [ 2024-09-13T12:02:06.123 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.124 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "take", 2024-09-13T12:02:06.124 INFO:tasks.workunit.client.0.smithi032.stdout: "item": -1, 2024-09-13T12:02:06.124 INFO:tasks.workunit.client.0.smithi032.stdout: "item_name": "default" 2024-09-13T12:02:06.124 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.124 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.124 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "choose_firstn", 2024-09-13T12:02:06.124 INFO:tasks.workunit.client.0.smithi032.stdout: "num": 0, 2024-09-13T12:02:06.124 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "osd" 2024-09-13T12:02:06.124 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.124 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.124 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "emit" 2024-09-13T12:02:06.124 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.124 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:02:06.124 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.125 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:02:06.125 INFO:tasks.workunit.client.0.smithi032.stdout: "tunables": { 2024-09-13T12:02:06.125 INFO:tasks.workunit.client.0.smithi032.stdout: "choose_local_tries": 0, 2024-09-13T12:02:06.125 INFO:tasks.workunit.client.0.smithi032.stdout: "choose_local_fallback_tries": 0, 2024-09-13T12:02:06.125 INFO:tasks.workunit.client.0.smithi032.stdout: "choose_total_tries": 50, 2024-09-13T12:02:06.125 INFO:tasks.workunit.client.0.smithi032.stdout: "chooseleaf_descend_once": 1, 2024-09-13T12:02:06.125 INFO:tasks.workunit.client.0.smithi032.stdout: "chooseleaf_vary_r": 1, 2024-09-13T12:02:06.125 INFO:tasks.workunit.client.0.smithi032.stdout: "chooseleaf_stable": 1, 2024-09-13T12:02:06.125 INFO:tasks.workunit.client.0.smithi032.stdout: "msr_descents": 100, 2024-09-13T12:02:06.125 INFO:tasks.workunit.client.0.smithi032.stdout: "msr_collision_tries": 100, 2024-09-13T12:02:06.125 INFO:tasks.workunit.client.0.smithi032.stdout: "straw_calc_version": 1, 2024-09-13T12:02:06.125 INFO:tasks.workunit.client.0.smithi032.stdout: "allowed_bucket_algs": 54, 2024-09-13T12:02:06.125 INFO:tasks.workunit.client.0.smithi032.stdout: "profile": "jewel", 2024-09-13T12:02:06.125 INFO:tasks.workunit.client.0.smithi032.stdout: "optimal_tunables": 1, 2024-09-13T12:02:06.125 INFO:tasks.workunit.client.0.smithi032.stdout: "legacy_tunables": 0, 2024-09-13T12:02:06.126 INFO:tasks.workunit.client.0.smithi032.stdout: "minimum_required_version": "jewel", 2024-09-13T12:02:06.126 INFO:tasks.workunit.client.0.smithi032.stdout: "require_feature_tunables": 1, 2024-09-13T12:02:06.126 INFO:tasks.workunit.client.0.smithi032.stdout: "require_feature_tunables2": 1, 2024-09-13T12:02:06.126 INFO:tasks.workunit.client.0.smithi032.stdout: "has_v2_rules": 0, 2024-09-13T12:02:06.126 INFO:tasks.workunit.client.0.smithi032.stdout: "require_feature_tunables3": 1, 2024-09-13T12:02:06.126 INFO:tasks.workunit.client.0.smithi032.stdout: "has_v3_rules": 0, 2024-09-13T12:02:06.126 INFO:tasks.workunit.client.0.smithi032.stdout: "has_v4_buckets": 1, 2024-09-13T12:02:06.126 INFO:tasks.workunit.client.0.smithi032.stdout: "require_feature_tunables5": 1, 2024-09-13T12:02:06.126 INFO:tasks.workunit.client.0.smithi032.stdout: "has_v5_rules": 0, 2024-09-13T12:02:06.126 INFO:tasks.workunit.client.0.smithi032.stdout: "has_msr_rules": 0 2024-09-13T12:02:06.126 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.126 INFO:tasks.workunit.client.0.smithi032.stdout: "choose_args": {} 2024-09-13T12:02:06.126 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.126 INFO:tasks.workunit.client.0.smithi032.stdout: "fsmap": { 2024-09-13T12:02:06.126 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 1, 2024-09-13T12:02:06.126 INFO:tasks.workunit.client.0.smithi032.stdout: "btime": "2024-09-13T11:58:41:300686+0000", 2024-09-13T12:02:06.127 INFO:tasks.workunit.client.0.smithi032.stdout: "default_fscid": -1, 2024-09-13T12:02:06.127 INFO:tasks.workunit.client.0.smithi032.stdout: "compat": { 2024-09-13T12:02:06.127 INFO:tasks.workunit.client.0.smithi032.stderr:report 2176941069 2024-09-13T12:02:06.127 INFO:tasks.workunit.client.0.smithi032.stdout: "compat": {}, 2024-09-13T12:02:06.127 INFO:tasks.workunit.client.0.smithi032.stdout: "ro_compat": {}, 2024-09-13T12:02:06.127 INFO:tasks.workunit.client.0.smithi032.stdout: "incompat": { 2024-09-13T12:02:06.127 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_1": "base v0.20", 2024-09-13T12:02:06.128 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_2": "client writeable ranges", 2024-09-13T12:02:06.128 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_3": "default file layouts on dirs", 2024-09-13T12:02:06.128 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_4": "dir inode in separate object", 2024-09-13T12:02:06.128 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_5": "mds uses versioned encoding", 2024-09-13T12:02:06.128 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-13T12:02:06.128 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_8": "no anchor table", 2024-09-13T12:02:06.128 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_9": "file layout v2", 2024-09-13T12:02:06.128 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_10": "snaprealm v2", 2024-09-13T12:02:06.128 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_11": "minor log segments", 2024-09-13T12:02:06.128 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_12": "quiesce subvolumes" 2024-09-13T12:02:06.128 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.128 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.128 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_flags": { 2024-09-13T12:02:06.128 INFO:tasks.workunit.client.0.smithi032.stdout: "enable_multiple": true, 2024-09-13T12:02:06.128 INFO:tasks.workunit.client.0.smithi032.stdout: "ever_enabled_multiple": true 2024-09-13T12:02:06.129 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.129 INFO:tasks.workunit.client.0.smithi032.stdout: "standbys": [], 2024-09-13T12:02:06.129 INFO:tasks.workunit.client.0.smithi032.stdout: "filesystems": [] 2024-09-13T12:02:06.129 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.129 INFO:tasks.workunit.client.0.smithi032.stdout: "mdsmap_first_committed": 1, 2024-09-13T12:02:06.129 INFO:tasks.workunit.client.0.smithi032.stdout: "mdsmap_last_committed": 1, 2024-09-13T12:02:06.129 INFO:tasks.workunit.client.0.smithi032.stdout: "auth": { 2024-09-13T12:02:06.129 INFO:tasks.workunit.client.0.smithi032.stdout: "first_committed": 1, 2024-09-13T12:02:06.129 INFO:tasks.workunit.client.0.smithi032.stdout: "last_committed": 5, 2024-09-13T12:02:06.129 INFO:tasks.workunit.client.0.smithi032.stdout: "num_secrets": 3 2024-09-13T12:02:06.129 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.129 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg": 1, 2024-09-13T12:02:06.130 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg_active": 1, 2024-09-13T12:02:06.130 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg_unknown": 0, 2024-09-13T12:02:06.130 INFO:tasks.workunit.client.0.smithi032.stdout: "num_osd": 3, 2024-09-13T12:02:06.130 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_sum": { 2024-09-13T12:02:06.130 INFO:tasks.workunit.client.0.smithi032.stdout: "stat_sum": { 2024-09-13T12:02:06.130 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes": 48300, 2024-09-13T12:02:06.130 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects": 100, 2024-09-13T12:02:06.130 INFO:tasks.workunit.client.0.smithi032.stdout: "num_object_clones": 0, 2024-09-13T12:02:06.130 INFO:tasks.workunit.client.0.smithi032.stdout: "num_object_copies": 300, 2024-09-13T12:02:06.130 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_missing_on_primary": 0, 2024-09-13T12:02:06.130 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_missing": 0, 2024-09-13T12:02:06.130 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_degraded": 100, 2024-09-13T12:02:06.130 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_misplaced": 0, 2024-09-13T12:02:06.130 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_unfound": 0, 2024-09-13T12:02:06.130 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_dirty": 100, 2024-09-13T12:02:06.131 INFO:tasks.workunit.client.0.smithi032.stdout: "num_whiteouts": 0, 2024-09-13T12:02:06.131 INFO:tasks.workunit.client.0.smithi032.stdout: "num_read": 0, 2024-09-13T12:02:06.131 INFO:tasks.workunit.client.0.smithi032.stdout: "num_read_kb": 0, 2024-09-13T12:02:06.131 INFO:tasks.workunit.client.0.smithi032.stdout: "num_write": 100, 2024-09-13T12:02:06.131 INFO:tasks.workunit.client.0.smithi032.stdout: "num_write_kb": 100, 2024-09-13T12:02:06.131 INFO:tasks.workunit.client.0.smithi032.stdout: "num_scrub_errors": 0, 2024-09-13T12:02:06.131 INFO:tasks.workunit.client.0.smithi032.stdout: "num_shallow_scrub_errors": 0, 2024-09-13T12:02:06.131 INFO:tasks.workunit.client.0.smithi032.stdout: "num_deep_scrub_errors": 0, 2024-09-13T12:02:06.131 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_recovered": 0, 2024-09-13T12:02:06.131 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes_recovered": 0, 2024-09-13T12:02:06.131 INFO:tasks.workunit.client.0.smithi032.stdout: "num_keys_recovered": 0, 2024-09-13T12:02:06.131 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_omap": 0, 2024-09-13T12:02:06.131 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_hit_set_archive": 0, 2024-09-13T12:02:06.131 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes_hit_set_archive": 0, 2024-09-13T12:02:06.131 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush": 0, 2024-09-13T12:02:06.131 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_kb": 0, 2024-09-13T12:02:06.132 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict": 0, 2024-09-13T12:02:06.132 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_kb": 0, 2024-09-13T12:02:06.132 INFO:tasks.workunit.client.0.smithi032.stdout: "num_promote": 0, 2024-09-13T12:02:06.132 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_mode_high": 0, 2024-09-13T12:02:06.132 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_mode_low": 0, 2024-09-13T12:02:06.132 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_mode_some": 0, 2024-09-13T12:02:06.132 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_mode_full": 0, 2024-09-13T12:02:06.132 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_pinned": 0, 2024-09-13T12:02:06.132 INFO:tasks.workunit.client.0.smithi032.stdout: "num_legacy_snapsets": 0, 2024-09-13T12:02:06.132 INFO:tasks.workunit.client.0.smithi032.stdout: "num_large_omap_objects": 0, 2024-09-13T12:02:06.132 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_manifest": 0, 2024-09-13T12:02:06.132 INFO:tasks.workunit.client.0.smithi032.stdout: "num_omap_bytes": 0, 2024-09-13T12:02:06.132 INFO:tasks.workunit.client.0.smithi032.stdout: "num_omap_keys": 0, 2024-09-13T12:02:06.132 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_repaired": 0 2024-09-13T12:02:06.132 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.133 INFO:tasks.workunit.client.0.smithi032.stdout: "store_stats": { 2024-09-13T12:02:06.133 INFO:tasks.workunit.client.0.smithi032.stdout: "total": 0, 2024-09-13T12:02:06.133 INFO:tasks.workunit.client.0.smithi032.stdout: "available": 0, 2024-09-13T12:02:06.133 INFO:tasks.workunit.client.0.smithi032.stdout: "internally_reserved": 0, 2024-09-13T12:02:06.133 INFO:tasks.workunit.client.0.smithi032.stdout: "allocated": 0, 2024-09-13T12:02:06.133 INFO:tasks.workunit.client.0.smithi032.stdout: "data_stored": 0, 2024-09-13T12:02:06.133 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed": 0, 2024-09-13T12:02:06.133 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_allocated": 0, 2024-09-13T12:02:06.133 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_original": 0, 2024-09-13T12:02:06.133 INFO:tasks.workunit.client.0.smithi032.stdout: "omap_allocated": 0, 2024-09-13T12:02:06.133 INFO:tasks.workunit.client.0.smithi032.stdout: "internal_metadata": 0 2024-09-13T12:02:06.134 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.134 INFO:tasks.workunit.client.0.smithi032.stdout: "log_size": 16, 2024-09-13T12:02:06.134 INFO:tasks.workunit.client.0.smithi032.stdout: "ondisk_log_size": 16, 2024-09-13T12:02:06.134 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 2, 2024-09-13T12:02:06.134 INFO:tasks.workunit.client.0.smithi032.stdout: "acting": 2, 2024-09-13T12:02:06.134 INFO:tasks.workunit.client.0.smithi032.stdout: "num_store_stats": 0 2024-09-13T12:02:06.134 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.134 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_sum": { 2024-09-13T12:02:06.134 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 0, 2024-09-13T12:02:06.135 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 0, 2024-09-13T12:02:06.135 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pgs": 3, 2024-09-13T12:02:06.136 INFO:tasks.workunit.client.0.smithi032.stdout: "num_osds": 3, 2024-09-13T12:02:06.136 INFO:tasks.workunit.client.0.smithi032.stdout: "num_per_pool_osds": 3, 2024-09-13T12:02:06.136 INFO:tasks.workunit.client.0.smithi032.stdout: "num_per_pool_omap_osds": 3, 2024-09-13T12:02:06.136 INFO:tasks.workunit.client.0.smithi032.stdout: "kb": 314572800, 2024-09-13T12:02:06.136 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used": 83636, 2024-09-13T12:02:06.136 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_data": 2088, 2024-09-13T12:02:06.136 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_omap": 10, 2024-09-13T12:02:06.136 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_meta": 81461, 2024-09-13T12:02:06.136 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_avail": 314489164, 2024-09-13T12:02:06.136 INFO:tasks.workunit.client.0.smithi032.stdout: "statfs": { 2024-09-13T12:02:06.136 INFO:tasks.workunit.client.0.smithi032.stdout: "total": 322122547200, 2024-09-13T12:02:06.136 INFO:tasks.workunit.client.0.smithi032.stdout: "available": 322036903936, 2024-09-13T12:02:06.136 INFO:tasks.workunit.client.0.smithi032.stdout: "internally_reserved": 0, 2024-09-13T12:02:06.136 INFO:tasks.workunit.client.0.smithi032.stdout: "allocated": 2138112, 2024-09-13T12:02:06.137 INFO:tasks.workunit.client.0.smithi032.stdout: "data_stored": 410796, 2024-09-13T12:02:06.137 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed": 0, 2024-09-13T12:02:06.137 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_allocated": 0, 2024-09-13T12:02:06.137 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_original": 0, 2024-09-13T12:02:06.137 INFO:tasks.workunit.client.0.smithi032.stdout: "omap_allocated": 10396, 2024-09-13T12:02:06.137 INFO:tasks.workunit.client.0.smithi032.stdout: "internal_metadata": 83416932 2024-09-13T12:02:06.137 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.137 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_peers": [], 2024-09-13T12:02:06.137 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_trim_queue_len": 0, 2024-09-13T12:02:06.137 INFO:tasks.workunit.client.0.smithi032.stdout: "num_snap_trimming": 0, 2024-09-13T12:02:06.137 INFO:tasks.workunit.client.0.smithi032.stdout: "num_shards_repaired": 1, 2024-09-13T12:02:06.137 INFO:tasks.workunit.client.0.smithi032.stdout: "op_queue_age_hist": { 2024-09-13T12:02:06.137 INFO:tasks.workunit.client.0.smithi032.stdout: "histogram": [], 2024-09-13T12:02:06.137 INFO:tasks.workunit.client.0.smithi032.stdout: "upper_bound": 1 2024-09-13T12:02:06.138 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.138 INFO:tasks.workunit.client.0.smithi032.stdout: "perf_stat": { 2024-09-13T12:02:06.138 INFO:tasks.workunit.client.0.smithi032.stdout: "commit_latency_ms": 0, 2024-09-13T12:02:06.138 INFO:tasks.workunit.client.0.smithi032.stdout: "apply_latency_ms": 0, 2024-09-13T12:02:06.138 INFO:tasks.workunit.client.0.smithi032.stdout: "commit_latency_ns": 0, 2024-09-13T12:02:06.138 INFO:tasks.workunit.client.0.smithi032.stdout: "apply_latency_ns": 0 2024-09-13T12:02:06.138 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.138 INFO:tasks.workunit.client.0.smithi032.stdout: "alerts": [], 2024-09-13T12:02:06.138 INFO:tasks.workunit.client.0.smithi032.stdout: "network_ping_times": [] 2024-09-13T12:02:06.138 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.138 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_sum_by_class": { 2024-09-13T12:02:06.138 INFO:tasks.workunit.client.0.smithi032.stdout: "hdd": { 2024-09-13T12:02:06.138 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 0, 2024-09-13T12:02:06.138 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 0, 2024-09-13T12:02:06.138 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pgs": 3, 2024-09-13T12:02:06.139 INFO:tasks.workunit.client.0.smithi032.stdout: "num_osds": 3, 2024-09-13T12:02:06.139 INFO:tasks.workunit.client.0.smithi032.stdout: "num_per_pool_osds": 3, 2024-09-13T12:02:06.139 INFO:tasks.workunit.client.0.smithi032.stdout: "num_per_pool_omap_osds": 3, 2024-09-13T12:02:06.139 INFO:tasks.workunit.client.0.smithi032.stdout: "kb": 314572800, 2024-09-13T12:02:06.139 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used": 83636, 2024-09-13T12:02:06.139 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_data": 2088, 2024-09-13T12:02:06.139 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_omap": 10, 2024-09-13T12:02:06.139 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_meta": 81461, 2024-09-13T12:02:06.139 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_avail": 314489164, 2024-09-13T12:02:06.139 INFO:tasks.workunit.client.0.smithi032.stdout: "statfs": { 2024-09-13T12:02:06.139 INFO:tasks.workunit.client.0.smithi032.stdout: "total": 322122547200, 2024-09-13T12:02:06.139 INFO:tasks.workunit.client.0.smithi032.stdout: "available": 322036903936, 2024-09-13T12:02:06.139 INFO:tasks.workunit.client.0.smithi032.stdout: "internally_reserved": 0, 2024-09-13T12:02:06.139 INFO:tasks.workunit.client.0.smithi032.stdout: "allocated": 2138112, 2024-09-13T12:02:06.139 INFO:tasks.workunit.client.0.smithi032.stdout: "data_stored": 410796, 2024-09-13T12:02:06.140 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed": 0, 2024-09-13T12:02:06.140 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_allocated": 0, 2024-09-13T12:02:06.140 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_original": 0, 2024-09-13T12:02:06.140 INFO:tasks.workunit.client.0.smithi032.stdout: "omap_allocated": 10396, 2024-09-13T12:02:06.140 INFO:tasks.workunit.client.0.smithi032.stdout: "internal_metadata": 83416932 2024-09-13T12:02:06.140 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.140 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_peers": [], 2024-09-13T12:02:06.140 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_trim_queue_len": 0, 2024-09-13T12:02:06.140 INFO:tasks.workunit.client.0.smithi032.stdout: "num_snap_trimming": 0, 2024-09-13T12:02:06.140 INFO:tasks.workunit.client.0.smithi032.stdout: "num_shards_repaired": 1, 2024-09-13T12:02:06.140 INFO:tasks.workunit.client.0.smithi032.stdout: "op_queue_age_hist": { 2024-09-13T12:02:06.140 INFO:tasks.workunit.client.0.smithi032.stdout: "histogram": [], 2024-09-13T12:02:06.140 INFO:tasks.workunit.client.0.smithi032.stdout: "upper_bound": 1 2024-09-13T12:02:06.140 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.140 INFO:tasks.workunit.client.0.smithi032.stdout: "perf_stat": { 2024-09-13T12:02:06.141 INFO:tasks.workunit.client.0.smithi032.stdout: "commit_latency_ms": 0, 2024-09-13T12:02:06.141 INFO:tasks.workunit.client.0.smithi032.stdout: "apply_latency_ms": 0, 2024-09-13T12:02:06.141 INFO:tasks.workunit.client.0.smithi032.stdout: "commit_latency_ns": 0, 2024-09-13T12:02:06.141 INFO:tasks.workunit.client.0.smithi032.stdout: "apply_latency_ns": 0 2024-09-13T12:02:06.141 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.141 INFO:tasks.workunit.client.0.smithi032.stdout: "alerts": [], 2024-09-13T12:02:06.141 INFO:tasks.workunit.client.0.smithi032.stdout: "network_ping_times": [] 2024-09-13T12:02:06.141 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.141 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.141 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_stats": [ 2024-09-13T12:02:06.141 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.141 INFO:tasks.workunit.client.0.smithi032.stdout: "poolid": 1, 2024-09-13T12:02:06.141 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg": 1, 2024-09-13T12:02:06.141 INFO:tasks.workunit.client.0.smithi032.stdout: "stat_sum": { 2024-09-13T12:02:06.141 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes": 48300, 2024-09-13T12:02:06.142 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects": 100, 2024-09-13T12:02:06.142 INFO:tasks.workunit.client.0.smithi032.stdout: "num_object_clones": 0, 2024-09-13T12:02:06.142 INFO:tasks.workunit.client.0.smithi032.stdout: "num_object_copies": 300, 2024-09-13T12:02:06.142 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_missing_on_primary": 0, 2024-09-13T12:02:06.142 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_missing": 0, 2024-09-13T12:02:06.142 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_degraded": 100, 2024-09-13T12:02:06.142 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_misplaced": 0, 2024-09-13T12:02:06.142 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_unfound": 0, 2024-09-13T12:02:06.142 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_dirty": 100, 2024-09-13T12:02:06.142 INFO:tasks.workunit.client.0.smithi032.stdout: "num_whiteouts": 0, 2024-09-13T12:02:06.142 INFO:tasks.workunit.client.0.smithi032.stdout: "num_read": 0, 2024-09-13T12:02:06.142 INFO:tasks.workunit.client.0.smithi032.stdout: "num_read_kb": 0, 2024-09-13T12:02:06.142 INFO:tasks.workunit.client.0.smithi032.stdout: "num_write": 100, 2024-09-13T12:02:06.142 INFO:tasks.workunit.client.0.smithi032.stdout: "num_write_kb": 100, 2024-09-13T12:02:06.142 INFO:tasks.workunit.client.0.smithi032.stdout: "num_scrub_errors": 0, 2024-09-13T12:02:06.143 INFO:tasks.workunit.client.0.smithi032.stdout: "num_shallow_scrub_errors": 0, 2024-09-13T12:02:06.143 INFO:tasks.workunit.client.0.smithi032.stdout: "num_deep_scrub_errors": 0, 2024-09-13T12:02:06.143 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_recovered": 0, 2024-09-13T12:02:06.143 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes_recovered": 0, 2024-09-13T12:02:06.143 INFO:tasks.workunit.client.0.smithi032.stdout: "num_keys_recovered": 0, 2024-09-13T12:02:06.143 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_omap": 0, 2024-09-13T12:02:06.143 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_hit_set_archive": 0, 2024-09-13T12:02:06.143 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes_hit_set_archive": 0, 2024-09-13T12:02:06.143 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush": 0, 2024-09-13T12:02:06.143 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_kb": 0, 2024-09-13T12:02:06.143 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict": 0, 2024-09-13T12:02:06.143 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_kb": 0, 2024-09-13T12:02:06.143 INFO:tasks.workunit.client.0.smithi032.stdout: "num_promote": 0, 2024-09-13T12:02:06.144 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_mode_high": 0, 2024-09-13T12:02:06.144 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_mode_low": 0, 2024-09-13T12:02:06.144 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_mode_some": 0, 2024-09-13T12:02:06.144 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_mode_full": 0, 2024-09-13T12:02:06.144 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_pinned": 0, 2024-09-13T12:02:06.144 INFO:tasks.workunit.client.0.smithi032.stdout: "num_legacy_snapsets": 0, 2024-09-13T12:02:06.144 INFO:tasks.workunit.client.0.smithi032.stdout: "num_large_omap_objects": 0, 2024-09-13T12:02:06.144 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_manifest": 0, 2024-09-13T12:02:06.144 INFO:tasks.workunit.client.0.smithi032.stdout: "num_omap_bytes": 0, 2024-09-13T12:02:06.144 INFO:tasks.workunit.client.0.smithi032.stdout: "num_omap_keys": 0, 2024-09-13T12:02:06.144 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_repaired": 0 2024-09-13T12:02:06.144 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.144 INFO:tasks.workunit.client.0.smithi032.stdout: "store_stats": { 2024-09-13T12:02:06.144 INFO:tasks.workunit.client.0.smithi032.stdout: "total": 0, 2024-09-13T12:02:06.145 INFO:tasks.workunit.client.0.smithi032.stdout: "available": 0, 2024-09-13T12:02:06.145 INFO:tasks.workunit.client.0.smithi032.stdout: "internally_reserved": 0, 2024-09-13T12:02:06.145 INFO:tasks.workunit.client.0.smithi032.stdout: "allocated": 1228800, 2024-09-13T12:02:06.145 INFO:tasks.workunit.client.0.smithi032.stdout: "data_stored": 144900, 2024-09-13T12:02:06.145 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed": 0, 2024-09-13T12:02:06.145 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_allocated": 0, 2024-09-13T12:02:06.145 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_original": 0, 2024-09-13T12:02:06.145 INFO:tasks.workunit.client.0.smithi032.stdout: "omap_allocated": 0, 2024-09-13T12:02:06.145 INFO:tasks.workunit.client.0.smithi032.stdout: "internal_metadata": 0 2024-09-13T12:02:06.145 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.146 INFO:tasks.workunit.client.0.smithi032.stdout: "log_size": 16, 2024-09-13T12:02:06.146 INFO:tasks.workunit.client.0.smithi032.stdout: "ondisk_log_size": 16, 2024-09-13T12:02:06.146 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 2, 2024-09-13T12:02:06.146 INFO:tasks.workunit.client.0.smithi032.stdout: "acting": 2, 2024-09-13T12:02:06.146 INFO:tasks.workunit.client.0.smithi032.stdout: "num_store_stats": 3 2024-09-13T12:02:06.146 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.146 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:02:06.146 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_stats": [ 2024-09-13T12:02:06.146 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.146 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 0, 2024-09-13T12:02:06.146 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 150323855371 2024-09-13T12:02:06.146 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.146 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.147 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 1, 2024-09-13T12:02:06.147 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 42949672987 2024-09-13T12:02:06.147 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.147 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.147 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2, 2024-09-13T12:02:06.147 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 158913789962 2024-09-13T12:02:06.147 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.147 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:02:06.147 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg_by_state": [ 2024-09-13T12:02:06.147 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.147 INFO:tasks.workunit.client.0.smithi032.stdout: "state": "active+undersized+degraded", 2024-09-13T12:02:06.147 INFO:tasks.workunit.client.0.smithi032.stdout: "num": 1 2024-09-13T12:02:06.147 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.147 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:02:06.147 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg_by_osd": [ 2024-09-13T12:02:06.148 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.148 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2, 2024-09-13T12:02:06.148 INFO:tasks.workunit.client.0.smithi032.stdout: "num_primary_pg": 0, 2024-09-13T12:02:06.148 INFO:tasks.workunit.client.0.smithi032.stdout: "num_acting_pg": 1, 2024-09-13T12:02:06.148 INFO:tasks.workunit.client.0.smithi032.stdout: "num_up_not_acting_pg": 0 2024-09-13T12:02:06.148 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.148 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.148 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 0, 2024-09-13T12:02:06.148 INFO:tasks.workunit.client.0.smithi032.stdout: "num_primary_pg": 1, 2024-09-13T12:02:06.148 INFO:tasks.workunit.client.0.smithi032.stdout: "num_acting_pg": 1, 2024-09-13T12:02:06.148 INFO:tasks.workunit.client.0.smithi032.stdout: "num_up_not_acting_pg": 0 2024-09-13T12:02:06.148 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.148 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.148 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 1, 2024-09-13T12:02:06.149 INFO:tasks.workunit.client.0.smithi032.stdout: "num_primary_pg": 0, 2024-09-13T12:02:06.149 INFO:tasks.workunit.client.0.smithi032.stdout: "num_acting_pg": 0, 2024-09-13T12:02:06.149 INFO:tasks.workunit.client.0.smithi032.stdout: "num_up_not_acting_pg": 0 2024-09-13T12:02:06.149 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.149 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:02:06.149 INFO:tasks.workunit.client.0.smithi032.stdout: "purged_snaps": [ 2024-09-13T12:02:06.149 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:02:06.149 INFO:tasks.workunit.client.0.smithi032.stdout: "pool": 1, 2024-09-13T12:02:06.149 INFO:tasks.workunit.client.0.smithi032.stdout: "purged_snaps": {} 2024-09-13T12:02:06.149 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.149 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:02:06.149 INFO:tasks.workunit.client.0.smithi032.stdout: "servicemap": { 2024-09-13T12:02:06.149 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 5, 2024-09-13T12:02:06.149 INFO:tasks.workunit.client.0.smithi032.stdout: "modified": "2024-09-13T12:01:26.272054+0000", 2024-09-13T12:02:06.149 INFO:tasks.workunit.client.0.smithi032.stdout: "services": { 2024-09-13T12:02:06.149 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": { 2024-09-13T12:02:06.150 INFO:tasks.workunit.client.0.smithi032.stdout: "daemons": { 2024-09-13T12:02:06.150 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": "", 2024-09-13T12:02:06.150 INFO:tasks.workunit.client.0.smithi032.stdout: "0": { 2024-09-13T12:02:06.150 INFO:tasks.workunit.client.0.smithi032.stdout: "start_epoch": 0, 2024-09-13T12:02:06.150 INFO:tasks.workunit.client.0.smithi032.stdout: "start_stamp": "0.000000", 2024-09-13T12:02:06.150 INFO:tasks.workunit.client.0.smithi032.stdout: "gid": 0, 2024-09-13T12:02:06.150 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "(unrecognized address family 0)/0", 2024-09-13T12:02:06.150 INFO:tasks.workunit.client.0.smithi032.stdout: "metadata": {}, 2024-09-13T12:02:06.150 INFO:tasks.workunit.client.0.smithi032.stdout: "task_status": {} 2024-09-13T12:02:06.150 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.150 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.150 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.150 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.150 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:02:06.151 INFO:tasks.workunit.client.0.smithi032.stdout: "mgrstat_first_committed": 1, 2024-09-13T12:02:06.151 INFO:tasks.workunit.client.0.smithi032.stdout: "mgrstat_last_committed": 91, 2024-09-13T12:02:06.151 INFO:tasks.workunit.client.0.smithi032.stdout: "logm_first_committed": 1, 2024-09-13T12:02:06.151 INFO:tasks.workunit.client.0.smithi032.stdout: "logm_last_committed": 385, 2024-09-13T12:02:06.151 INFO:tasks.workunit.client.0.smithi032.stdout: "paxos": { 2024-09-13T12:02:06.151 INFO:tasks.workunit.client.0.smithi032.stdout: "first_committed": 1, 2024-09-13T12:02:06.151 INFO:tasks.workunit.client.0.smithi032.stdout: "last_committed": 522, 2024-09-13T12:02:06.151 INFO:tasks.workunit.client.0.smithi032.stdout: "last_pn": 0, 2024-09-13T12:02:06.151 INFO:tasks.workunit.client.0.smithi032.stdout: "accepted_pn": 0 2024-09-13T12:02:06.151 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:02:06.151 INFO:tasks.workunit.client.0.smithi032.stdout:} 2024-09-13T12:02:06.152 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-13T12:02:06.152 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2024-09-13T12:02:06.152 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2024-09-13T12:02:06.152 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2024-09-13T12:02:06.152 INFO:tasks.workunit.client.0.smithi032.stdout:writing non-divergent object existing_4 2024-09-13T12:02:06.152 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2024-09-13T12:02:06.487 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:02:06.487 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 100 0 100 0 0 48300 0 0 16 84 16 active+undersized+degraded 2024-09-13T12:01:28.780872+0000 28'100 40:125 [0,2] 0 [0,2] 0 0'0 2024-09-13T12:00:27.843316+0000 0'0 2024-09-13T12:00:27.843316+0000 0 0 periodic scrub scheduled @ 2024-09-14T15:23:57.457221+0000 0 0 2024-09-13T12:02:06.487 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:02:06.487 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:02:06.487 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:02:06.498 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2024-09-13T12:02:06.609 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2024-09-13T12:02:06.609 INFO:tasks.workunit.client.0.smithi032.stdout:delay recovery 2024-09-13T12:02:06.609 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2024-09-13T12:02:06.999 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:02:06.999 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 100 0 100 0 0 48300 0 0 16 84 16 active+undersized+degraded 2024-09-13T12:01:28.780872+0000 28'100 40:125 [0,2] 0 [0,2] 0 0'0 2024-09-13T12:00:27.843316+0000 0'0 2024-09-13T12:00:27.843316+0000 0 0 periodic scrub scheduled @ 2024-09-14T15:23:57.457221+0000 0 0 2024-09-13T12:02:06.999 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:02:06.999 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:02:06.999 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:02:07.010 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-09-13T12:02:07.011 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2024-09-13T12:02:07.011 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-13T12:02:07.011 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-13T12:02:07.011 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:02:07.011 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:02:07.011 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:02:07.011 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-osd.0.asok 2024-09-13T12:02:07.011 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-09-13T12:02:07.012 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.33242/ceph-osd.0.asok set_recovery_delay 100000 2024-09-13T12:02:07.095 INFO:tasks.workunit.client.0.smithi032.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-13T12:02:07.102 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-09-13T12:02:07.103 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2024-09-13T12:02:07.103 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-13T12:02:07.104 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-13T12:02:07.104 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:02:07.104 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:02:07.104 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:02:07.104 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-osd.2.asok 2024-09-13T12:02:07.104 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-09-13T12:02:07.104 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.33242/ceph-osd.2.asok set_recovery_delay 100000 2024-09-13T12:02:07.188 INFO:tasks.workunit.client.0.smithi032.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-13T12:02:07.195 INFO:tasks.workunit.client.0.smithi032.stdout:revive divergent 1 2024-09-13T12:02:07.195 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2024-09-13T12:02:07.195 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2024-09-13T12:02:07.620 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:02:07.620 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 100 0 100 0 0 48300 0 0 16 84 16 active+undersized+degraded 2024-09-13T12:01:28.780872+0000 28'100 40:125 [0,2] 0 [0,2] 0 0'0 2024-09-13T12:00:27.843316+0000 0'0 2024-09-13T12:00:27.843316+0000 0 0 periodic scrub scheduled @ 2024-09-14T15:23:57.457221+0000 0 0 2024-09-13T12:02:07.620 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:02:07.620 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:02:07.620 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:02:07.630 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2024-09-13T12:02:08.250 INFO:tasks.workunit.client.0.smithi032.stderr:noup is set 2024-09-13T12:02:08.265 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2024-09-13T12:02:08.265 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-13T12:02:08.265 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:02:08.265 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T12:02:08.265 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:02:08.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-13T12:02:08.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T12:02:08.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:02:08.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:02:08.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:02:08.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-13T12:02:08.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-13T12:02:08.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:02:08.267 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:02:08.267 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T12:02:08.267 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:02:08.267 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:02:08.267 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:02:08.267 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:02:08.267 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:02:08.267 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:02:08.267 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:02:08.267 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:02:08.267 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:02:08.268 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T12:02:08.268 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T12:02:08.268 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:02:08.268 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:02:08.268 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:02:08.268 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:02:08.268 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:02:08.268 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:02:08.268 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-13T12:02:08.269 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.1 2024-09-13T12:02:08.269 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T12:02:08.270 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:02:08.270 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-13T12:02:08.270 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T12:02:08.270 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:02:08.270 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:02:08.270 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:02:08.302 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:02:08.301+0000 7f7324d32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:02:08.337 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:02:08.336+0000 7f7324d32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:02:08.338 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:02:08.337+0000 7f7324d32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:02:08.687 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: return 0 2024-09-13T12:02:08.687 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2024-09-13T12:02:09.676 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:02:09.675+0000 7f7324d32640 -1 Falling back to public interface 2024-09-13T12:02:12.653 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:02:12.652+0000 7f7324d32640 -1 osd.1 28 log_to_monitors true 2024-09-13T12:02:13.688 INFO:tasks.workunit.client.0.smithi032.stdout:delay recovery divergent 2024-09-13T12:02:13.688 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2024-09-13T12:02:13.688 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2024-09-13T12:02:14.090 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:02:14.090 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-13T12:01:28.780872+0000 40'101 42:129 [0,2] 0 [0,2] 0 0'0 2024-09-13T12:00:27.843316+0000 0'0 2024-09-13T12:00:27.843316+0000 0 0 periodic scrub scheduled @ 2024-09-14T15:23:57.457221+0000 0 0 2024-09-13T12:02:14.090 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:02:14.090 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:02:14.090 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:02:14.102 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2024-09-13T12:02:14.102 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-13T12:02:14.102 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-13T12:02:14.103 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:02:14.103 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:02:14.103 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:02:14.103 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-osd.1.asok 2024-09-13T12:02:14.103 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2024-09-13T12:02:14.103 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.33242/ceph-osd.1.asok set_recovery_delay 100000 2024-09-13T12:02:14.187 INFO:tasks.workunit.client.0.smithi032.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-13T12:02:14.195 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2024-09-13T12:02:14.797 INFO:tasks.workunit.client.0.smithi032.stderr:noup is unset 2024-09-13T12:02:14.809 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2024-09-13T12:02:14.809 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:02:14.810 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T12:02:14.810 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:02:14.810 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:02:14.810 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:02:14.810 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:02:14.810 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:02:14.810 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:02:14.811 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:02:15.314 INFO:tasks.workunit.client.0.smithi032.stdout:osd.0 up in weight 1 up_from 35 up_thru 39 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/442043230,v1:127.0.0.1:6803/442043230] [v2:127.0.0.1:6804/442043230,v1:127.0.0.1:6805/442043230] exists,up 42e90f08-426a-4b5b-ad78-7bd0b8144f62 2024-09-13T12:02:15.315 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:02:15.315 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:02:15.315 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:02:15.315 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2024-09-13T12:02:15.315 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:02:15.315 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T12:02:15.315 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:02:15.315 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:02:15.315 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:02:15.315 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:02:15.316 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:02:15.316 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:02:15.316 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:02:15.723 INFO:tasks.workunit.client.0.smithi032.stdout:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/1361522729,v1:127.0.0.1:6819/1361522729] [v2:127.0.0.1:6820/1361522729,v1:127.0.0.1:6821/1361522729] exists,up 45eb455b-ec14-492a-a47e-4e0a84ac4a8c 2024-09-13T12:02:15.724 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:02:15.724 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:02:15.724 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:02:15.725 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2024-09-13T12:02:15.725 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:02:15.725 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T12:02:15.725 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:02:15.725 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:02:15.725 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:02:15.725 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:02:15.725 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:02:15.726 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:02:15.726 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:02:16.137 INFO:tasks.workunit.client.0.smithi032.stdout:osd.2 up in weight 1 up_from 37 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/3448235176,v1:127.0.0.1:6811/3448235176] [v2:127.0.0.1:6812/3448235176,v1:127.0.0.1:6813/3448235176] exists,up 4607d722-ea41-4ddb-a0cd-5570afca8326 2024-09-13T12:02:16.138 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:02:16.138 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:02:16.138 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:02:16.138 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2024-09-13T12:02:16.528 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:02:16.528 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-13T12:01:28.780872+0000 40'101 42:129 [0,2] 0 [0,2] 0 0'0 2024-09-13T12:00:27.843316+0000 0'0 2024-09-13T12:00:27.843316+0000 0 0 periodic scrub scheduled @ 2024-09-14T15:23:57.457221+0000 0 0 2024-09-13T12:02:16.528 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:02:16.528 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:02:16.528 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:02:16.538 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2024-09-13T12:02:16.539 INFO:tasks.workunit.client.0.smithi032.stdout:wait for peering 2024-09-13T12:02:16.539 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2024-09-13T12:02:16.926 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:02:16.926 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-13T12:01:28.780872+0000 40'101 42:129 [0,2] 0 [0,2] 0 0'0 2024-09-13T12:00:27.843316+0000 0'0 2024-09-13T12:00:27.843316+0000 0 0 periodic scrub scheduled @ 2024-09-14T15:23:57.457221+0000 0 0 2024-09-13T12:02:16.926 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:02:16.926 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:02:16.926 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:02:16.937 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2024-09-13T12:02:17.065 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2024-09-13T12:02:17.065 INFO:tasks.workunit.client.0.smithi032.stdout:killing divergent 1 2024-09-13T12:02:17.065 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2024-09-13T12:02:17.457 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:02:17.458 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-13T12:01:28.780872+0000 40'101 42:129 [0,2] 0 [0,2] 0 0'0 2024-09-13T12:00:27.843316+0000 0'0 2024-09-13T12:00:27.843316+0000 0 0 periodic scrub scheduled @ 2024-09-14T15:23:57.457221+0000 0 0 2024-09-13T12:02:17.458 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:02:17.458 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:02:17.458 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:02:17.469 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2024-09-13T12:02:17.469 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:02:17.470 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:02:17.470 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:02:17.470 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:02:17.470 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:02:17.574 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:02:17.575 INFO:tasks.workunit.client.0.smithi032.stdout:reviving divergent 1 2024-09-13T12:02:17.575 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2024-09-13T12:02:17.575 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2024-09-13T12:02:17.968 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:02:17.969 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-13T12:01:28.780872+0000 40'101 42:129 [0,2] 0 [0,2] 0 0'0 2024-09-13T12:00:27.843316+0000 0'0 2024-09-13T12:00:27.843316+0000 0 0 periodic scrub scheduled @ 2024-09-14T15:23:57.457221+0000 0 0 2024-09-13T12:02:17.969 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:02:17.969 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:02:17.969 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:02:17.982 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2024-09-13T12:02:17.982 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-13T12:02:17.982 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:02:17.983 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T12:02:17.983 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:02:17.983 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-13T12:02:17.983 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T12:02:17.983 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:02:17.983 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:02:17.983 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:02:17.983 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-13T12:02:17.983 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-13T12:02:17.983 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:02:17.983 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:02:17.983 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T12:02:17.983 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:02:17.983 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:02:17.984 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:02:17.984 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:02:17.984 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:02:17.984 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:02:17.984 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:02:17.984 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:02:17.984 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:02:17.984 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T12:02:17.984 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T12:02:17.984 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:02:17.984 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:02:17.984 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:02:17.985 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:02:17.985 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:02:17.985 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:02:17.985 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-13T12:02:17.985 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.1 2024-09-13T12:02:17.986 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T12:02:17.986 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:02:17.986 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-13T12:02:17.986 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T12:02:17.987 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:02:17.987 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:02:17.987 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:02:18.018 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:02:18.017+0000 7f0c43683640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:02:18.020 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:02:18.019+0000 7f0c43683640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:02:18.021 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:02:18.020+0000 7f0c43683640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:02:18.397 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T12:02:18.397 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:02:18.397 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T12:02:18.397 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:02:18.397 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:02:18.397 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:02:18.397 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:02:18.398 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:02:18.398 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:02:18.398 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:02:18.846 INFO:tasks.workunit.client.0.smithi032.stdout:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/1361522729,v1:127.0.0.1:6819/1361522729] [v2:127.0.0.1:6820/1361522729,v1:127.0.0.1:6821/1361522729] exists,up 45eb455b-ec14-492a-a47e-4e0a84ac4a8c 2024-09-13T12:02:18.847 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:02:18.847 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:02:18.847 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:02:18.847 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2024-09-13T12:02:19.386 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:02:19.385+0000 7f0c43683640 -1 Falling back to public interface 2024-09-13T12:02:22.052 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:02:22.051+0000 7f0c43683640 -1 osd.1 46 log_to_monitors true 2024-09-13T12:02:38.848 INFO:tasks.workunit.client.0.smithi032.stdout:allowing recovery 2024-09-13T12:02:38.848 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2024-09-13T12:02:38.849 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2024-09-13T12:02:39.253 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:02:39.253 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 101 0 0 0 0 48454 0 0 18 84 18 active+clean 2024-09-13T12:02:23.405384+0000 46'102 50:154 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-13T12:00:27.843316+0000 0'0 2024-09-13T12:00:27.843316+0000 0 0 periodic scrub scheduled @ 2024-09-14T19:01:38.945041+0000 0 0 2024-09-13T12:02:39.254 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:02:39.254 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:02:39.254 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:02:39.265 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-09-13T12:02:39.265 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-13T12:02:39.382 INFO:tasks.workunit.client.0.smithi032.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-13T12:02:39.391 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-09-13T12:02:39.391 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-13T12:02:39.506 INFO:tasks.workunit.client.0.smithi032.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-13T12:02:39.515 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-09-13T12:02:39.515 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-13T12:02:39.630 INFO:tasks.workunit.client.0.smithi032.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-13T12:02:39.640 INFO:tasks.workunit.client.0.smithi032.stdout:reading divergent objects 2024-09-13T12:02:39.640 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2024-09-13T12:02:39.640 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2024-09-13T12:02:40.030 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:02:40.030 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 101 0 0 0 0 48454 0 0 18 84 18 active+clean 2024-09-13T12:02:23.405384+0000 46'102 50:154 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-13T12:00:27.843316+0000 0'0 2024-09-13T12:00:27.843316+0000 0 0 periodic scrub scheduled @ 2024-09-14T19:01:38.945041+0000 0 0 2024-09-13T12:02:40.030 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:02:40.030 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:02:40.030 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:02:40.041 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2024-09-13T12:02:40.042 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2024-09-13T12:02:40.043 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-13T12:02:40.043 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2024-09-13T12:02:40.081 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-13T12:02:40.081 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2024-09-13T12:02:40.118 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-13T12:02:40.118 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2024-09-13T12:02:40.155 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-13T12:02:40.155 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2024-09-13T12:02:40.192 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2024-09-13T12:02:40.193 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-09-13T12:02:40.194 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-13T12:02:40.200 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:02:15.301+0000 7f7308ca4640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4622.0:1 2024-09-13T12:01:02.322075+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-13T12:02:40.200 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:02:15.301+0000 7f7308ca4640 20 _merge_object_divergent_entries: keeping 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4622.0:1 2024-09-13T12:01:02.322075+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-13T12:02:40.200 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:02:15.301+0000 7f7308ca4640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-13T12:02:40.200 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:02:15.301+0000 7f7308ca4640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 28'100 first_divergent_update: 28'101 last_divergent_update: 28'101 2024-09-13T12:02:40.200 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:02:15.301+0000 7f7308ca4640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-09-13T12:02:40.200 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:02:15.301+0000 7f7308ca4640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-09-13T12:02:40.200 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:02:15.301+0000 7f7308ca4640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 28'101 2024-09-13T12:02:40.200 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:02:15.301+0000 7f7308ca4640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4622.0:1 2024-09-13T12:01:02.322075+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-13T12:02:40.200 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:02:15.301+0000 7f7308ca4640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-09-13T12:02:40.201 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-09-13T12:02:40.201 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-13T12:02:40.205 INFO:tasks.workunit.client.0.smithi032.stdout:success 2024-09-13T12:02:40.206 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2024-09-13T12:02:40.206 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2024-09-13T12:02:40.206 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test 2024-09-13T12:02:40.206 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-13T12:02:40.830 INFO:tasks.workunit.client.0.smithi032.stderr:pool 'test' does not exist 2024-09-13T12:02:40.843 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2024-09-13T12:02:40.843 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:02:40.843 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:02:40.843 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:02:40.843 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:02:40.843 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:02:41.152 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:02:41.153 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-13T12:02:41.153 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-13T12:02:41.153 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T12:02:41.153 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-13T12:02:41.153 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:02:41.153 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:02:41.153 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:02:41.153 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:02:41.153 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:02:41.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:02:41.158 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T12:02:41.159 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T12:02:41.159 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T12:02:41.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T12:02:41.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T12:02:41.160 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T12:02:41.161 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:02:41.161 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T12:02:41.162 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T12:02:41.162 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:02:41.163 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:02:41.164 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T12:02:41.164 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-13T12:02:41.184 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T12:02:41.184 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:02:41.184 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:02:41.184 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.33242 2024-09-13T12:02:41.186 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T12:02:41.186 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T12:02:41.186 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-13T12:02:41.186 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-13T12:02:41.186 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/divergent-priors 2024-09-13T12:02:41.187 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/divergent-priors 2024-09-13T12:02:41.187 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-13T12:02:41.187 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T12:02:41.187 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-13T12:02:41.187 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:02:41.187 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:02:41.187 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:02:41.187 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:02:41.187 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:02:41.187 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:02:41.188 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T12:02:41.189 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T12:02:41.189 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T12:02:41.190 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T12:02:41.190 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T12:02:41.190 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T12:02:41.191 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:02:41.191 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T12:02:41.192 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T12:02:41.192 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:02:41.192 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:02:41.193 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T12:02:41.193 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-13T12:02:41.195 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T12:02:41.195 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:02:41.195 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:02:41.195 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.33242 2024-09-13T12:02:41.195 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T12:02:41.196 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T12:02:41.196 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/divergent-priors 2024-09-13T12:02:41.197 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T12:02:41.197 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:02:41.197 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:02:41.197 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.33242 2024-09-13T12:02:41.199 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T12:02:41.199 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T12:02:41.199 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T12:02:41.199 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-13T12:02:41.199 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2024-09-13T12:02:41.199 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2024-09-13T12:02:41.199 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2024-09-13T12:02:41.200 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2024-09-13T12:02:41.200 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2024-09-13T12:02:41.200 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2024-09-13T12:02:41.200 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2024-09-13T12:02:41.202 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2024-09-13T12:02:41.202 INFO:tasks.workunit.client.0.smithi032.stderr:1 2024-09-13T12:02:41.202 INFO:tasks.workunit.client.0.smithi032.stderr:2' 2024-09-13T12:02:41.202 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2024-09-13T12:02:41.202 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/divergent-priors 2024-09-13T12:02:41.202 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T12:02:41.202 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T12:02:41.202 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T12:02:41.202 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/divergent-priors/a 2024-09-13T12:02:41.202 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-13T12:02:41.714 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T12:02:41.714 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:02:41.714 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:02:41.715 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:02:41.715 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:02:41.715 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:02:41.715 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:02:41.715 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-13T12:02:41.914 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T12:02:41.916 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T12:02:41.916 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T12:02:41.916 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T12:02:41.916 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T12:02:41.916 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T12:02:41.916 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T12:02:41.916 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T12:02:41.917 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T12:02:41.917 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:02:41.917 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:02:41.917 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:02:41.917 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-mon.a.asok 2024-09-13T12:02:41.917 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T12:02:41.917 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.33242/ceph-mon.a.asok config get fsid 2024-09-13T12:02:41.999 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T12:02:41.999 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T12:02:41.999 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T12:02:41.999 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T12:02:41.999 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T12:02:41.999 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T12:02:41.999 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T12:02:41.999 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T12:02:42.000 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:02:42.000 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:02:42.000 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:02:42.000 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-mon.a.asok 2024-09-13T12:02:42.000 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T12:02:42.000 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.33242/ceph-mon.a.asok config get mon_host 2024-09-13T12:02:42.080 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2024-09-13T12:02:42.081 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/divergent-priors 2024-09-13T12:02:42.081 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T12:02:42.081 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T12:02:42.081 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T12:02:42.081 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/divergent-priors/x 2024-09-13T12:02:42.081 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T12:02:42.948 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T12:02:42.948 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:02:42.948 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:02:42.948 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:02:42.948 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:02:42.948 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:02:42.948 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:02:42.949 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T12:02:42.949 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T12:02:42.986 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-09-13T12:02:42.986 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2024-09-13T12:02:42.986 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-13T12:02:42.986 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T12:02:42.987 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T12:02:42.987 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T12:02:42.987 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/0 2024-09-13T12:02:42.987 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T12:02:42.987 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:02:42.987 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:02:42.987 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:02:42.987 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-13T12:02:42.987 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-13T12:02:42.987 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T12:02:42.987 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T12:02:42.987 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T12:02:42.987 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T12:02:42.988 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:02:42.988 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:02:42.988 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:02:42.988 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:02:42.988 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:02:42.988 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:02:42.988 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:02:42.988 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:02:42.988 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T12:02:42.988 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T12:02:42.988 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T12:02:42.988 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T12:02:42.988 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:02:42.989 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:02:42.989 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:02:42.989 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:02:42.989 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T12:02:42.989 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T12:02:42.989 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/0 2024-09-13T12:02:42.989 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T12:02:42.990 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=940cd0ac-d018-458e-a8bb-6eb19147770c 2024-09-13T12:02:42.990 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 940cd0ac-d018-458e-a8bb-6eb19147770c' 2024-09-13T12:02:42.990 INFO:tasks.workunit.client.0.smithi032.stdout:add osd0 940cd0ac-d018-458e-a8bb-6eb19147770c 2024-09-13T12:02:42.990 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T12:02:43.013 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDjKeRmNUG0ABAADdof9BfeWgCC7LdX6GEKuQ== 2024-09-13T12:02:43.013 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDjKeRmNUG0ABAADdof9BfeWgCC7LdX6GEKuQ=="}' 2024-09-13T12:02:43.013 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 940cd0ac-d018-458e-a8bb-6eb19147770c -i td/divergent-priors/0/new.json 2024-09-13T12:02:43.289 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:02:43.298 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/0/new.json 2024-09-13T12:02:43.299 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDjKeRmNUG0ABAADdof9BfeWgCC7LdX6GEKuQ== --osd-uuid 940cd0ac-d018-458e-a8bb-6eb19147770c 2024-09-13T12:02:43.337 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:02:43.336+0000 7fd9fb2e0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:02:43.349 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:02:43.348+0000 7fd9fb2e0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:02:43.350 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:02:43.349+0000 7fd9fb2e0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:02:43.350 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:02:43.349+0000 7fd9fb2e0640 -1 bdev(0x556916335c00 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-09-13T12:02:43.350 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:02:43.349+0000 7fd9fb2e0640 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-13T12:02:49.094 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-13T12:02:49.094 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T12:02:49.095 INFO:tasks.workunit.client.0.smithi032.stdout:adding osd0 key to auth repository 2024-09-13T12:02:49.095 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T12:02:49.095 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T12:02:49.681 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.0 2024-09-13T12:02:49.682 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T12:02:49.682 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:02:49.682 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T12:02:49.682 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T12:02:49.682 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T12:02:49.717 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:02:49.716+0000 7fbac8fa8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:02:49.719 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:02:49.718+0000 7fbac8fa8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:02:49.720 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:02:49.719+0000 7fbac8fa8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:02:50.132 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T12:02:50.132 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:02:50.132 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T12:02:50.132 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:02:50.132 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:02:50.132 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:02:50.133 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:02:50.133 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:02:50.133 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:02:50.133 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:02:50.540 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:02:50.812 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:02:50.811+0000 7fbac8fa8640 -1 Falling back to public interface 2024-09-13T12:02:51.542 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:02:51.542 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:02:51.542 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:02:51.543 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:02:51.543 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:02:51.543 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:02:51.992 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:02:52.993 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:02:52.993 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:02:52.993 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:02:52.994 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:02:52.994 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:02:52.994 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:02:53.418 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:02:53.554 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:02:53.553+0000 7fbac8fa8640 -1 osd.0 0 log_to_monitors true 2024-09-13T12:02:54.420 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:02:54.420 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:02:54.420 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:02:54.421 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:02:54.421 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:02:54.421 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:02:54.874 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:02:55.875 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:02:55.875 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:02:55.875 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:02:55.876 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:02:55.876 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:02:55.876 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:02:56.335 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:02:57.336 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:02:57.337 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:02:57.337 INFO:tasks.workunit.client.0.smithi032.stdout:5 2024-09-13T12:02:57.337 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T12:02:57.337 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:02:57.338 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:02:57.758 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:02:58.760 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:02:58.760 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:02:58.760 INFO:tasks.workunit.client.0.smithi032.stdout:6 2024-09-13T12:02:58.761 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T12:02:58.761 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:02:58.761 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:02:59.184 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:03:00.186 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:03:00.186 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:03:00.186 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T12:03:00.186 INFO:tasks.workunit.client.0.smithi032.stdout:7 2024-09-13T12:03:00.187 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:03:00.187 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:03:00.713 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:03:01.716 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:03:01.716 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:03:01.716 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T12:03:01.716 INFO:tasks.workunit.client.0.smithi032.stdout:8 2024-09-13T12:03:01.717 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:03:01.717 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:03:02.143 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:03:03.144 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:03:03.145 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:03:03.145 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T12:03:03.148 INFO:tasks.workunit.client.0.smithi032.stdout:9 2024-09-13T12:03:03.148 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:03:03.148 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:03:03.590 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:03:04.592 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:03:04.592 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:03:04.592 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T12:03:04.593 INFO:tasks.workunit.client.0.smithi032.stdout:10 2024-09-13T12:03:04.593 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:03:04.593 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:03:05.113 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:03:06.114 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:03:06.114 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:03:06.115 INFO:tasks.workunit.client.0.smithi032.stdout:11 2024-09-13T12:03:06.115 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T12:03:06.115 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:03:06.115 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:03:06.602 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:03:07.603 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:03:07.604 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:03:07.604 INFO:tasks.workunit.client.0.smithi032.stdout:12 2024-09-13T12:03:07.604 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T12:03:07.604 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:03:07.605 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:03:08.304 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:03:09.306 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:03:09.306 INFO:tasks.workunit.client.0.smithi032.stdout:13 2024-09-13T12:03:09.307 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:03:09.307 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T12:03:09.307 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:03:09.307 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:03:10.533 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:03:11.535 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:03:11.535 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:03:11.535 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T12:03:11.535 INFO:tasks.workunit.client.0.smithi032.stdout:14 2024-09-13T12:03:11.536 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:03:11.536 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:03:12.032 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:03:13.033 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:03:13.033 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:03:13.033 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T12:03:13.034 INFO:tasks.workunit.client.0.smithi032.stdout:15 2024-09-13T12:03:13.034 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:03:13.034 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:03:13.586 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:03:14.588 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:03:14.588 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:03:14.588 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T12:03:14.588 INFO:tasks.workunit.client.0.smithi032.stdout:16 2024-09-13T12:03:14.589 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:03:14.589 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:03:15.437 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:03:16.438 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:03:16.438 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:03:16.438 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T12:03:16.439 INFO:tasks.workunit.client.0.smithi032.stdout:17 2024-09-13T12:03:16.439 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:03:16.439 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:03:16.909 INFO:tasks.workunit.client.0.smithi032.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/976199658,v1:127.0.0.1:6803/976199658] [v2:127.0.0.1:6804/976199658,v1:127.0.0.1:6805/976199658] exists,up 940cd0ac-d018-458e-a8bb-6eb19147770c 2024-09-13T12:03:16.913 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:03:16.913 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:03:16.913 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:03:16.913 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-09-13T12:03:16.913 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2024-09-13T12:03:16.913 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-13T12:03:16.913 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T12:03:16.914 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T12:03:16.914 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T12:03:16.914 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/1 2024-09-13T12:03:16.914 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T12:03:16.914 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:03:16.914 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:03:16.914 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:03:16.914 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-13T12:03:16.914 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-13T12:03:16.915 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T12:03:16.915 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T12:03:16.915 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T12:03:16.915 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T12:03:16.915 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:03:16.915 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:03:16.915 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:03:16.915 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:03:16.915 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:03:16.915 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:03:16.915 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:03:16.915 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:03:16.915 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T12:03:16.916 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T12:03:16.916 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T12:03:16.916 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T12:03:16.916 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:03:16.916 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:03:16.916 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:03:16.916 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:03:16.916 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T12:03:16.916 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T12:03:16.916 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/1 2024-09-13T12:03:16.916 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T12:03:16.916 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=eb913216-75c2-4cc2-ad3b-855fbbad0fc3 2024-09-13T12:03:16.917 INFO:tasks.workunit.client.0.smithi032.stdout:add osd1 eb913216-75c2-4cc2-ad3b-855fbbad0fc3 2024-09-13T12:03:16.917 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 eb913216-75c2-4cc2-ad3b-855fbbad0fc3' 2024-09-13T12:03:16.917 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T12:03:16.938 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAEKuRmdQXjNxAAAm3CEXYnmrys/kF7nMBD+w== 2024-09-13T12:03:16.939 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAEKuRmdQXjNxAAAm3CEXYnmrys/kF7nMBD+w=="}' 2024-09-13T12:03:16.939 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new eb913216-75c2-4cc2-ad3b-855fbbad0fc3 -i td/divergent-priors/1/new.json 2024-09-13T12:03:17.430 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:03:17.441 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/1/new.json 2024-09-13T12:03:17.442 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAEKuRmdQXjNxAAAm3CEXYnmrys/kF7nMBD+w== --osd-uuid eb913216-75c2-4cc2-ad3b-855fbbad0fc3 2024-09-13T12:03:17.480 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:03:17.478+0000 7f0fdca31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:03:17.482 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:03:17.480+0000 7f0fdca31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:03:17.483 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:03:17.481+0000 7f0fdca31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:03:17.483 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:03:17.482+0000 7f0fdca31640 -1 bdev(0x557e4aa89c00 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-09-13T12:03:17.484 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:03:17.482+0000 7f0fdca31640 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-13T12:03:23.863 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-13T12:03:23.864 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T12:03:23.865 INFO:tasks.workunit.client.0.smithi032.stdout:adding osd1 key to auth repository 2024-09-13T12:03:23.865 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T12:03:23.865 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T12:03:24.445 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.1 2024-09-13T12:03:24.445 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T12:03:24.446 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:03:24.446 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T12:03:24.446 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T12:03:24.446 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T12:03:24.480 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:03:24.478+0000 7f42967cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:03:24.482 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:03:24.481+0000 7f42967cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:03:24.484 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:03:24.482+0000 7f42967cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:03:24.900 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T12:03:24.900 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:03:24.900 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T12:03:24.900 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:03:24.900 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:03:24.900 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:03:24.900 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:03:24.901 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:03:24.901 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:03:24.901 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:03:25.342 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:03:26.066 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:03:26.064+0000 7f42967cc640 -1 Falling back to public interface 2024-09-13T12:03:26.343 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:03:26.344 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:03:26.344 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:03:26.344 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:03:26.344 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:03:26.344 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:03:26.802 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:03:27.803 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:03:27.804 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:03:27.804 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:03:27.804 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:03:27.805 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:03:27.805 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:03:28.257 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:03:28.865 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:03:28.864+0000 7f42967cc640 -1 osd.1 0 log_to_monitors true 2024-09-13T12:03:29.258 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:03:29.258 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:03:29.258 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:03:29.259 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:03:29.259 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:03:29.259 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:03:29.798 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:03:30.800 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:03:30.800 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:03:30.800 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:03:30.800 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:03:30.801 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:03:30.801 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:03:31.255 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:03:32.257 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:03:32.257 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:03:32.257 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T12:03:32.257 INFO:tasks.workunit.client.0.smithi032.stdout:5 2024-09-13T12:03:32.258 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:03:32.258 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:03:32.799 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:03:33.801 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:03:33.801 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:03:33.801 INFO:tasks.workunit.client.0.smithi032.stdout:6 2024-09-13T12:03:33.802 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T12:03:33.802 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:03:33.802 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:03:34.291 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:03:35.296 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:03:35.297 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:03:35.297 INFO:tasks.workunit.client.0.smithi032.stdout:7 2024-09-13T12:03:35.297 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T12:03:35.297 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:03:35.297 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:03:35.806 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:03:36.807 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:03:36.808 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:03:36.808 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T12:03:36.808 INFO:tasks.workunit.client.0.smithi032.stdout:8 2024-09-13T12:03:36.808 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:03:36.810 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:03:37.231 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:03:38.233 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:03:38.233 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:03:38.233 INFO:tasks.workunit.client.0.smithi032.stdout:9 2024-09-13T12:03:38.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T12:03:38.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:03:38.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:03:38.735 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:03:39.737 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:03:39.737 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:03:39.737 INFO:tasks.workunit.client.0.smithi032.stdout:10 2024-09-13T12:03:39.738 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T12:03:39.738 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:03:39.738 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:03:40.161 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:03:41.163 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:03:41.163 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:03:41.163 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T12:03:41.163 INFO:tasks.workunit.client.0.smithi032.stdout:11 2024-09-13T12:03:41.164 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:03:41.164 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:03:41.995 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:03:42.996 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:03:42.997 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:03:42.997 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T12:03:42.997 INFO:tasks.workunit.client.0.smithi032.stdout:12 2024-09-13T12:03:42.997 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:03:42.997 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:03:43.762 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:03:44.763 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:03:44.763 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:03:44.764 INFO:tasks.workunit.client.0.smithi032.stdout:13 2024-09-13T12:03:44.764 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T12:03:44.764 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:03:44.764 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:03:45.506 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:03:46.507 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:03:46.507 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:03:46.508 INFO:tasks.workunit.client.0.smithi032.stdout:14 2024-09-13T12:03:46.508 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T12:03:46.508 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:03:46.508 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:03:47.250 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:03:48.251 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:03:48.252 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:03:48.252 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T12:03:48.252 INFO:tasks.workunit.client.0.smithi032.stdout:15 2024-09-13T12:03:48.253 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:03:48.253 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:03:48.993 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:03:49.995 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:03:49.995 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:03:49.995 INFO:tasks.workunit.client.0.smithi032.stdout:16 2024-09-13T12:03:49.996 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T12:03:49.996 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:03:49.996 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:03:50.816 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:03:51.818 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:03:51.818 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:03:51.818 INFO:tasks.workunit.client.0.smithi032.stdout:17 2024-09-13T12:03:51.819 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T12:03:51.819 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:03:51.819 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:03:52.312 INFO:tasks.workunit.client.0.smithi032.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1355014695,v1:127.0.0.1:6811/1355014695] [v2:127.0.0.1:6812/1355014695,v1:127.0.0.1:6813/1355014695] exists,up eb913216-75c2-4cc2-ad3b-855fbbad0fc3 2024-09-13T12:03:52.316 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:03:52.316 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:03:52.316 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:03:52.316 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-09-13T12:03:52.316 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2024-09-13T12:03:52.316 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-13T12:03:52.316 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T12:03:52.316 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T12:03:52.317 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T12:03:52.317 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/2 2024-09-13T12:03:52.317 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T12:03:52.317 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:03:52.317 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:03:52.317 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:03:52.317 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-13T12:03:52.317 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-13T12:03:52.317 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T12:03:52.317 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T12:03:52.317 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T12:03:52.317 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T12:03:52.318 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:03:52.318 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:03:52.318 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:03:52.318 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:03:52.318 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:03:52.318 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:03:52.318 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:03:52.318 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:03:52.318 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T12:03:52.318 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T12:03:52.318 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T12:03:52.318 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T12:03:52.318 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:03:52.319 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:03:52.319 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:03:52.319 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:03:52.319 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T12:03:52.319 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T12:03:52.319 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/2 2024-09-13T12:03:52.319 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T12:03:52.319 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=69901422-110f-4bc1-8cb2-72564b47a69b 2024-09-13T12:03:52.319 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 69901422-110f-4bc1-8cb2-72564b47a69b' 2024-09-13T12:03:52.320 INFO:tasks.workunit.client.0.smithi032.stdout:add osd2 69901422-110f-4bc1-8cb2-72564b47a69b 2024-09-13T12:03:52.320 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T12:03:52.340 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAoKuRmG/I8FBAAzAWo9wEnUuORzO66LFB4BQ== 2024-09-13T12:03:52.340 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAoKuRmG/I8FBAAzAWo9wEnUuORzO66LFB4BQ=="}' 2024-09-13T12:03:52.341 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 69901422-110f-4bc1-8cb2-72564b47a69b -i td/divergent-priors/2/new.json 2024-09-13T12:03:52.952 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:03:52.964 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/2/new.json 2024-09-13T12:03:52.965 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAoKuRmG/I8FBAAzAWo9wEnUuORzO66LFB4BQ== --osd-uuid 69901422-110f-4bc1-8cb2-72564b47a69b 2024-09-13T12:03:53.002 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:03:53.002+0000 7f88c1813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:03:53.010 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:03:53.010+0000 7f88c1813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:03:53.012 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:03:53.011+0000 7f88c1813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:03:53.012 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:03:53.011+0000 7f88c1813640 -1 bdev(0x55f6775f5c00 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-09-13T12:03:53.012 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:03:53.011+0000 7f88c1813640 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-13T12:03:59.384 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-13T12:03:59.384 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T12:03:59.385 INFO:tasks.workunit.client.0.smithi032.stdout:adding osd2 key to auth repository 2024-09-13T12:03:59.385 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T12:03:59.385 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T12:03:59.955 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.2 2024-09-13T12:03:59.956 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T12:03:59.956 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:03:59.956 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T12:03:59.956 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T12:03:59.956 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T12:03:59.991 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:03:59.990+0000 7f94b8558640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:03:59.993 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:03:59.992+0000 7f94b8558640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:03:59.994 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:03:59.993+0000 7f94b8558640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:04:00.375 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T12:04:00.375 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:04:00.375 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T12:04:00.375 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:04:00.375 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:04:00.375 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:04:00.375 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:04:00.376 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:04:00.376 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:04:00.376 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:04:00.856 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:04:01.364 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:04:01.363+0000 7f94b8558640 -1 Falling back to public interface 2024-09-13T12:04:01.857 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:04:01.858 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:04:01.858 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:04:01.858 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:04:01.859 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:04:01.859 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:04:02.308 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:04:03.309 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:04:03.309 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:04:03.310 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:04:03.310 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:04:03.310 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:04:03.310 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:04:03.740 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:04:03.878 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:04:03.876+0000 7f94b8558640 -1 osd.2 0 log_to_monitors true 2024-09-13T12:04:04.742 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:04:04.742 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:04:04.742 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:04:04.742 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:04:04.742 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:04:04.743 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:04:05.281 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:04:06.282 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:04:06.282 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:04:06.282 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:04:06.283 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:04:06.283 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:04:06.283 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:04:06.752 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:04:07.753 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:04:07.753 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:04:07.754 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T12:04:07.754 INFO:tasks.workunit.client.0.smithi032.stdout:5 2024-09-13T12:04:07.754 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:04:07.754 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:04:08.323 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:04:09.324 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:04:09.324 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:04:09.324 INFO:tasks.workunit.client.0.smithi032.stdout:6 2024-09-13T12:04:09.325 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T12:04:09.325 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:04:09.325 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:04:09.770 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:04:10.771 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:04:10.771 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:04:10.772 INFO:tasks.workunit.client.0.smithi032.stdout:7 2024-09-13T12:04:10.772 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T12:04:10.772 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:04:10.772 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:04:11.204 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:04:12.205 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:04:12.206 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:04:12.206 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T12:04:12.206 INFO:tasks.workunit.client.0.smithi032.stdout:8 2024-09-13T12:04:12.207 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:04:12.207 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:04:12.626 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:04:13.627 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:04:13.628 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:04:13.628 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T12:04:13.628 INFO:tasks.workunit.client.0.smithi032.stdout:9 2024-09-13T12:04:13.628 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:04:13.628 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:04:14.135 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:04:15.137 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:04:15.137 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:04:15.137 INFO:tasks.workunit.client.0.smithi032.stdout:10 2024-09-13T12:04:15.138 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T12:04:15.138 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:04:15.138 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:04:15.561 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:04:16.563 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:04:16.563 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:04:16.563 INFO:tasks.workunit.client.0.smithi032.stdout:11 2024-09-13T12:04:16.564 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T12:04:16.564 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:04:16.564 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:04:17.324 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:04:18.326 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:04:18.326 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:04:18.326 INFO:tasks.workunit.client.0.smithi032.stdout:12 2024-09-13T12:04:18.327 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T12:04:18.327 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:04:18.327 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:04:19.072 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:04:20.074 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:04:20.074 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:04:20.074 INFO:tasks.workunit.client.0.smithi032.stdout:13 2024-09-13T12:04:20.075 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T12:04:20.075 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:04:20.075 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:04:20.686 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:04:21.687 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:04:21.687 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:04:21.687 INFO:tasks.workunit.client.0.smithi032.stdout:14 2024-09-13T12:04:21.688 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T12:04:21.688 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:04:21.688 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:04:22.414 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:04:23.415 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:04:23.415 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:04:23.415 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T12:04:23.416 INFO:tasks.workunit.client.0.smithi032.stdout:15 2024-09-13T12:04:23.416 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:04:23.416 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:04:24.161 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:04:25.159 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:04:25.159 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:04:25.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T12:04:25.160 INFO:tasks.workunit.client.0.smithi032.stdout:16 2024-09-13T12:04:25.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:04:25.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:04:25.819 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:04:26.821 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:04:26.821 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:04:26.821 INFO:tasks.workunit.client.0.smithi032.stdout:17 2024-09-13T12:04:26.822 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T12:04:26.822 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:04:26.822 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:04:27.252 INFO:tasks.workunit.client.0.smithi032.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2224170215,v1:127.0.0.1:6819/2224170215] [v2:127.0.0.1:6820/2224170215,v1:127.0.0.1:6821/2224170215] exists,up 69901422-110f-4bc1-8cb2-72564b47a69b 2024-09-13T12:04:27.253 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:04:27.253 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:04:27.253 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:04:27.253 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2024-09-13T12:04:28.142 INFO:tasks.workunit.client.0.smithi032.stderr:noout is set 2024-09-13T12:04:28.153 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2024-09-13T12:04:28.794 INFO:tasks.workunit.client.0.smithi032.stderr:noin is set 2024-09-13T12:04:28.806 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2024-09-13T12:04:29.590 INFO:tasks.workunit.client.0.smithi032.stderr:nodown is set 2024-09-13T12:04:29.602 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2024-09-13T12:04:29.602 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-13T12:04:30.429 INFO:tasks.workunit.client.0.smithi032.stderr:pool 'test' already exists 2024-09-13T12:04:30.441 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T12:04:31.442 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2024-09-13T12:04:32.095 INFO:tasks.workunit.client.0.smithi032.stderr:set pool 1 size to 3 2024-09-13T12:04:32.106 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2024-09-13T12:04:32.882 INFO:tasks.workunit.client.0.smithi032.stderr:set pool 1 min_size to 2 2024-09-13T12:04:32.894 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2024-09-13T12:04:32.894 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T12:04:32.894 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T12:04:33.318 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T12:04:33.318 INFO:tasks.workunit.client.0.smithi032.stderr:1 2024-09-13T12:04:33.318 INFO:tasks.workunit.client.0.smithi032.stderr:2' 2024-09-13T12:04:33.318 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T12:04:33.318 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:04:33.318 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T12:04:33.442 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836501 2024-09-13T12:04:33.442 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836501 2024-09-13T12:04:33.442 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501' 2024-09-13T12:04:33.442 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:04:33.442 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T12:04:33.566 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672979 2024-09-13T12:04:33.567 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672979 2024-09-13T12:04:33.567 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672979' 2024-09-13T12:04:33.567 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:04:33.567 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T12:04:33.692 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509446 2024-09-13T12:04:33.692 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509446 2024-09-13T12:04:33.692 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672979 2-64424509446' 2024-09-13T12:04:33.692 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:04:33.692 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836501 2024-09-13T12:04:33.692 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:04:33.693 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T12:04:33.694 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836501 2024-09-13T12:04:33.694 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:04:33.694 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836501 2024-09-13T12:04:33.695 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.0 seq 21474836501 2024-09-13T12:04:33.695 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-13T12:04:33.695 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:04:34.134 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836499 -lt 21474836501 2024-09-13T12:04:34.134 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T12:04:35.135 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T12:04:35.136 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:04:35.594 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836501 2024-09-13T12:04:35.594 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:04:35.594 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672979 2024-09-13T12:04:35.594 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:04:35.595 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T12:04:35.595 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672979 2024-09-13T12:04:35.596 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:04:35.596 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672979 2024-09-13T12:04:35.597 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.1 seq 42949672979 2024-09-13T12:04:35.597 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-13T12:04:35.597 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T12:04:36.045 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672979 -lt 42949672979 2024-09-13T12:04:36.045 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:04:36.046 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509446 2024-09-13T12:04:36.046 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:04:36.046 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T12:04:36.047 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509446 2024-09-13T12:04:36.047 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:04:36.048 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509446 2024-09-13T12:04:36.048 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.2 seq 64424509446 2024-09-13T12:04:36.049 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-09-13T12:04:36.049 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T12:04:36.489 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509446 -lt 64424509446 2024-09-13T12:04:36.489 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2024-09-13T12:04:36.489 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T12:04:36.489 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T12:04:36.489 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T12:04:36.489 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T12:04:36.490 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T12:04:36.490 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T12:04:36.490 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T12:04:36.490 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T12:04:36.490 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T12:04:36.579 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T12:04:36.580 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T12:04:36.580 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T12:04:36.580 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T12:04:36.580 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T12:04:36.580 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T12:04:37.023 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T12:04:37.023 INFO:tasks.workunit.client.0.smithi032.stderr:1 2024-09-13T12:04:37.024 INFO:tasks.workunit.client.0.smithi032.stderr:2' 2024-09-13T12:04:37.024 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T12:04:37.024 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:04:37.024 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T12:04:37.148 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836504 2024-09-13T12:04:37.148 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836504 2024-09-13T12:04:37.148 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504' 2024-09-13T12:04:37.148 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:04:37.148 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T12:04:37.281 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672981 2024-09-13T12:04:37.281 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672981 2024-09-13T12:04:37.281 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672981' 2024-09-13T12:04:37.281 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:04:37.281 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T12:04:37.405 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509448 2024-09-13T12:04:37.405 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509448 2024-09-13T12:04:37.405 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672981 2-64424509448' 2024-09-13T12:04:37.406 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:04:37.406 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836504 2024-09-13T12:04:37.406 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:04:37.407 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T12:04:37.407 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836504 2024-09-13T12:04:37.407 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:04:37.408 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836504 2024-09-13T12:04:37.408 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.0 seq 21474836504 2024-09-13T12:04:37.409 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-13T12:04:37.409 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:04:37.850 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836504 2024-09-13T12:04:37.850 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T12:04:38.851 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T12:04:38.851 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:04:39.277 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836504 2024-09-13T12:04:39.277 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:04:39.278 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672981 2024-09-13T12:04:39.278 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:04:39.278 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T12:04:39.279 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672981 2024-09-13T12:04:39.279 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:04:39.280 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672981 2024-09-13T12:04:39.280 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.1 seq 42949672981 2024-09-13T12:04:39.281 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-09-13T12:04:39.281 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T12:04:39.728 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672982 -lt 42949672981 2024-09-13T12:04:39.728 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:04:39.729 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509448 2024-09-13T12:04:39.729 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:04:39.729 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T12:04:39.730 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509448 2024-09-13T12:04:39.730 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:04:39.731 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509448 2024-09-13T12:04:39.731 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-13T12:04:39.731 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.2 seq 64424509448 2024-09-13T12:04:39.731 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T12:04:40.162 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509449 -lt 64424509448 2024-09-13T12:04:40.162 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T12:04:40.163 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:04:40.163 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:04:40.714 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T12:04:40.715 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:04:40.715 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:04:40.715 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:04:40.715 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:04:40.715 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:04:40.715 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:04:40.715 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:04:41.151 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T12:04:41.151 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:04:41.151 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:04:41.151 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:04:41.705 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T12:04:41.705 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T12:04:41.705 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T12:04:41.705 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2024-09-13T12:04:41.706 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2024-09-13T12:04:42.109 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:04:42.120 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2024-09-13T12:04:42.121 INFO:tasks.workunit.client.0.smithi032.stdout:primary and soon to be divergent is 1 2024-09-13T12:04:42.121 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2024-09-13T12:04:42.121 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2024-09-13T12:04:42.512 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:04:42.512 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T12:04:31.124688+0000 0'0 29:22 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-13T12:04:30.066569+0000 0'0 2024-09-13T12:04:30.066569+0000 0 0 periodic scrub scheduled @ 2024-09-14T16:54:34.063909+0000 0 0 2024-09-13T12:04:42.513 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:04:42.513 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:04:42.513 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:04:42.524 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2024-09-13T12:04:42.524 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-09-13T12:04:42.524 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2024-09-13T12:04:42.524 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2024-09-13T12:04:42.524 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-09-13T12:04:42.525 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2024-09-13T12:04:42.525 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2024-09-13T12:04:42.525 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-09-13T12:04:42.525 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2024-09-13T12:04:42.525 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2024-09-13T12:04:42.525 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2024-09-13T12:04:42.525 INFO:tasks.workunit.client.0.smithi032.stdout:writing initial objects 2024-09-13T12:04:42.526 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2024-09-13T12:04:42.526 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:42.526 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2024-09-13T12:04:42.690 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:42.690 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2024-09-13T12:04:42.849 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:42.850 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2024-09-13T12:04:43.000 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:43.000 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2024-09-13T12:04:43.176 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:43.176 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2024-09-13T12:04:43.377 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:43.377 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2024-09-13T12:04:43.536 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:43.536 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2024-09-13T12:04:43.703 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:43.703 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2024-09-13T12:04:43.854 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:43.854 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2024-09-13T12:04:44.013 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:44.013 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2024-09-13T12:04:44.189 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:44.189 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2024-09-13T12:04:44.365 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:44.365 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2024-09-13T12:04:44.516 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:44.516 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2024-09-13T12:04:44.667 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:44.667 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2024-09-13T12:04:44.826 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:44.826 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2024-09-13T12:04:45.010 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:45.010 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2024-09-13T12:04:45.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:45.161 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2024-09-13T12:04:45.320 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:45.320 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2024-09-13T12:04:45.479 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:45.479 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2024-09-13T12:04:45.638 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:45.638 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2024-09-13T12:04:45.824 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:45.824 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2024-09-13T12:04:45.972 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:45.973 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2024-09-13T12:04:46.149 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:46.149 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2024-09-13T12:04:46.325 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:46.325 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2024-09-13T12:04:46.476 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:46.476 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2024-09-13T12:04:46.626 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:46.626 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2024-09-13T12:04:46.777 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:46.777 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2024-09-13T12:04:46.936 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:46.936 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2024-09-13T12:04:47.095 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:47.095 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2024-09-13T12:04:47.263 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:47.263 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2024-09-13T12:04:47.447 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:47.447 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2024-09-13T12:04:47.597 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:47.598 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2024-09-13T12:04:47.909 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:47.909 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2024-09-13T12:04:48.092 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:48.092 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2024-09-13T12:04:48.284 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:48.285 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2024-09-13T12:04:48.452 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:48.452 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2024-09-13T12:04:48.602 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:48.602 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2024-09-13T12:04:48.753 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:48.753 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2024-09-13T12:04:48.920 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:48.921 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2024-09-13T12:04:49.088 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:49.088 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2024-09-13T12:04:49.280 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:49.281 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2024-09-13T12:04:49.440 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:49.440 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2024-09-13T12:04:49.607 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:49.608 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2024-09-13T12:04:49.767 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:49.767 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2024-09-13T12:04:49.926 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:49.926 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2024-09-13T12:04:50.093 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:50.093 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2024-09-13T12:04:50.278 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:50.278 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2024-09-13T12:04:50.453 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:50.453 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2024-09-13T12:04:50.621 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:50.621 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2024-09-13T12:04:50.780 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:50.780 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2024-09-13T12:04:50.939 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:50.939 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2024-09-13T12:04:51.106 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:51.106 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2024-09-13T12:04:51.290 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:51.290 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2024-09-13T12:04:51.443 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:51.444 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2024-09-13T12:04:51.602 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:51.602 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2024-09-13T12:04:51.778 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:51.778 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2024-09-13T12:04:51.955 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:51.955 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2024-09-13T12:04:52.130 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:52.130 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2024-09-13T12:04:52.315 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:52.315 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2024-09-13T12:04:52.482 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:52.482 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2024-09-13T12:04:52.649 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:52.649 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2024-09-13T12:04:52.811 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:52.811 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2024-09-13T12:04:52.993 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:52.993 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2024-09-13T12:04:53.161 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:53.161 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2024-09-13T12:04:53.328 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:53.328 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2024-09-13T12:04:53.504 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:53.504 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2024-09-13T12:04:53.671 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:53.671 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2024-09-13T12:04:53.830 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:53.830 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2024-09-13T12:04:53.998 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:53.998 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2024-09-13T12:04:54.182 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:54.182 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2024-09-13T12:04:54.349 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:54.349 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2024-09-13T12:04:54.517 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:54.517 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2024-09-13T12:04:54.676 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:54.676 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2024-09-13T12:04:54.841 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:54.842 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2024-09-13T12:04:55.028 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:55.028 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2024-09-13T12:04:55.212 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:55.212 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2024-09-13T12:04:55.388 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:55.388 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2024-09-13T12:04:55.547 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:55.547 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2024-09-13T12:04:55.714 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:55.714 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2024-09-13T12:04:55.881 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:55.881 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2024-09-13T12:04:56.049 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:56.049 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2024-09-13T12:04:56.242 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:56.242 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2024-09-13T12:04:56.409 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:56.409 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2024-09-13T12:04:56.560 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:56.560 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2024-09-13T12:04:56.711 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:56.711 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2024-09-13T12:04:56.878 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:56.878 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2024-09-13T12:04:57.062 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:57.063 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2024-09-13T12:04:57.230 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:57.230 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2024-09-13T12:04:57.397 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:57.397 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2024-09-13T12:04:57.556 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:57.556 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2024-09-13T12:04:57.715 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:57.715 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2024-09-13T12:04:57.883 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:57.883 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2024-09-13T12:04:58.050 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:58.050 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2024-09-13T12:04:58.243 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:58.243 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2024-09-13T12:04:58.394 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:58.394 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2024-09-13T12:04:58.561 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:58.562 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2024-09-13T12:04:58.729 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:58.729 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2024-09-13T12:04:58.888 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:58.888 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2024-09-13T12:04:59.064 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:59.064 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2024-09-13T12:04:59.239 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-13T12:04:59.239 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2024-09-13T12:04:59.399 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-13T12:04:59.399 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2024-09-13T12:04:59.399 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T12:04:59.399 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T12:04:59.400 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T12:04:59.400 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-13T12:04:59.400 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T12:04:59.400 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T12:04:59.400 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T12:04:59.401 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T12:04:59.401 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T12:04:59.458 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-13T12:04:59.458 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T12:04:59.458 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T12:04:59.458 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T12:04:59.459 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T12:04:59.459 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T12:04:59.894 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T12:04:59.894 INFO:tasks.workunit.client.0.smithi032.stderr:1 2024-09-13T12:04:59.894 INFO:tasks.workunit.client.0.smithi032.stderr:2' 2024-09-13T12:04:59.894 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T12:04:59.894 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:04:59.894 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T12:05:00.018 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836510 2024-09-13T12:05:00.018 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836510 2024-09-13T12:05:00.018 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510' 2024-09-13T12:05:00.018 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:05:00.018 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T12:05:00.159 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672988 2024-09-13T12:05:00.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672988 2024-09-13T12:05:00.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-42949672988' 2024-09-13T12:05:00.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:05:00.160 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T12:05:00.285 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509455 2024-09-13T12:05:00.286 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509455 2024-09-13T12:05:00.286 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-42949672988 2-64424509455' 2024-09-13T12:05:00.286 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:05:00.286 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836510 2024-09-13T12:05:00.286 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:05:00.286 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T12:05:00.287 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836510 2024-09-13T12:05:00.287 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:05:00.288 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836510 2024-09-13T12:05:00.288 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.0 seq 21474836510 2024-09-13T12:05:00.289 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-09-13T12:05:00.289 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:05:00.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836508 -lt 21474836510 2024-09-13T12:05:00.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T12:05:01.721 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T12:05:01.722 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:05:02.163 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836511 -lt 21474836510 2024-09-13T12:05:02.163 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:05:02.163 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672988 2024-09-13T12:05:02.163 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:05:02.164 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T12:05:02.165 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672988 2024-09-13T12:05:02.165 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:05:02.166 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672988 2024-09-13T12:05:02.166 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.1 seq 42949672988 2024-09-13T12:05:02.166 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2024-09-13T12:05:02.166 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T12:05:02.606 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672988 -lt 42949672988 2024-09-13T12:05:02.606 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:05:02.606 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509455 2024-09-13T12:05:02.606 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:05:02.607 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T12:05:02.608 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509455 2024-09-13T12:05:02.608 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:05:02.609 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509455 2024-09-13T12:05:02.609 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.2 seq 64424509455 2024-09-13T12:05:02.609 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-13T12:05:02.610 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T12:05:03.057 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509455 -lt 64424509455 2024-09-13T12:05:03.058 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T12:05:03.058 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:05:03.058 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:05:03.610 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T12:05:03.610 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:05:03.610 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:05:03.610 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:05:03.610 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:05:03.610 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:05:03.610 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:05:03.610 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:05:04.026 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T12:05:04.027 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:05:04.027 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:05:04.027 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:05:04.642 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T12:05:04.642 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T12:05:04.642 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T12:05:04.642 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2024-09-13T12:05:04.642 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=test 2024-09-13T12:05:04.642 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=existing_1 2024-09-13T12:05:04.643 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map test existing_1 2024-09-13T12:05:04.643 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T12:05:05.095 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2024-09-13T12:05:05.095 INFO:tasks.workunit.client.0.smithi032.stdout:blackholing osds 0 2 2024-09-13T12:05:05.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2024-09-13T12:05:05.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2024-09-13T12:05:05.483 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:05:05.484 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 100 0 0 0 0 48300 0 0 16 84 16 active+clean 2024-09-13T12:04:31.124688+0000 29'100 29:122 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-13T12:04:30.066569+0000 0'0 2024-09-13T12:04:30.066569+0000 0 0 periodic scrub scheduled @ 2024-09-14T16:54:34.063909+0000 0 0 2024-09-13T12:05:05.484 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:05:05.484 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:05:05.484 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:05:05.494 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-09-13T12:05:05.496 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2024-09-13T12:05:05.496 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-13T12:05:05.496 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-13T12:05:05.496 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:05:05.496 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:05:05.496 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:05:05.496 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-osd.0.asok 2024-09-13T12:05:05.497 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-09-13T12:05:05.497 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.33242/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-13T12:05:05.576 INFO:tasks.workunit.client.0.smithi032.stdout:{ 2024-09-13T12:05:05.576 INFO:tasks.workunit.client.0.smithi032.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-13T12:05:05.576 INFO:tasks.workunit.client.0.smithi032.stdout:} 2024-09-13T12:05:05.583 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-09-13T12:05:05.584 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2024-09-13T12:05:05.584 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-13T12:05:05.584 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-13T12:05:05.584 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:05:05.584 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:05:05.584 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:05:05.584 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-osd.2.asok 2024-09-13T12:05:05.585 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-09-13T12:05:05.585 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.33242/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-13T12:05:05.664 INFO:tasks.workunit.client.0.smithi032.stdout:{ 2024-09-13T12:05:05.665 INFO:tasks.workunit.client.0.smithi032.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-13T12:05:05.665 INFO:tasks.workunit.client.0.smithi032.stdout:} 2024-09-13T12:05:05.671 INFO:tasks.workunit.client.0.smithi032.stdout:create new divergent objects 2024-09-13T12:05:05.672 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2024-09-13T12:05:05.672 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2024-09-13T12:05:05.673 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-13T12:05:05.673 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-13T12:05:05.673 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2024-09-13T12:05:05.673 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2024-09-13T12:05:05.673 INFO:tasks.workunit.client.0.smithi032.stdout:writing divergent objects 2024-09-13T12:05:05.674 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2024-09-13T12:05:05.674 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2024-09-13T12:05:05.674 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-13T12:05:05.674 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-13T12:05:05.674 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2024-09-13T12:05:05.675 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2024-09-13T12:05:05.675 INFO:tasks.workunit.client.0.smithi032.stdout:remove divergent objects 2024-09-13T12:05:05.675 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2024-09-13T12:05:05.675 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2024-09-13T12:05:05.676 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-13T12:05:05.676 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2024-09-13T12:05:05.677 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2024-09-13T12:05:05.678 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-13T12:05:05.678 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2024-09-13T12:05:05.678 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2024-09-13T12:05:05.680 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2024-09-13T12:05:05.680 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2024-09-13T12:05:05.680 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2024-09-13T12:05:15.682 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2024-09-13T12:05:15.687 INFO:tasks.workunit.client.0.smithi032.stderr:rados: no process found 2024-09-13T12:05:15.687 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2024-09-13T12:05:15.687 INFO:tasks.workunit.client.0.smithi032.stdout:killing all the osds 2024-09-13T12:05:15.687 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2024-09-13T12:05:16.093 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:05:16.093 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 100 0 0 0 0 46676 0 0 16 90 16 active+clean 2024-09-13T12:04:31.124688+0000 29'106 29:128 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-13T12:04:30.066569+0000 0'0 2024-09-13T12:04:30.066569+0000 0 0 periodic scrub scheduled @ 2024-09-14T16:54:34.063909+0000 0 0 2024-09-13T12:05:16.093 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:05:16.093 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:05:16.093 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:05:16.104 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2024-09-13T12:05:16.105 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:05:16.105 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:05:16.105 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:05:16.105 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:05:16.105 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:05:16.211 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:05:16.211 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-09-13T12:05:16.211 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2024-09-13T12:05:16.724 INFO:tasks.workunit.client.0.smithi032.stderr:osd.0 is already down. 2024-09-13T12:05:16.735 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-09-13T12:05:16.735 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2024-09-13T12:05:17.331 INFO:tasks.workunit.client.0.smithi032.stderr:osd.1 is already down. 2024-09-13T12:05:17.343 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-09-13T12:05:17.343 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2024-09-13T12:05:17.863 INFO:tasks.workunit.client.0.smithi032.stderr:osd.2 is already down. 2024-09-13T12:05:17.876 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-09-13T12:05:17.876 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2024-09-13T12:05:18.561 INFO:tasks.workunit.client.0.smithi032.stderr:osd.0 is already out. 2024-09-13T12:05:18.572 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-09-13T12:05:18.572 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2024-09-13T12:05:19.097 INFO:tasks.workunit.client.0.smithi032.stderr:osd.2 is already out. 2024-09-13T12:05:19.108 INFO:tasks.workunit.client.0.smithi032.stdout:bringing up non_divergent 0 2 2024-09-13T12:05:19.109 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2024-09-13T12:05:19.109 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2024-09-13T12:05:19.500 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:05:19.500 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 100 0 0 0 0 46676 0 0 16 90 16 stale+active+clean 2024-09-13T12:04:31.124688+0000 29'106 29:128 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-13T12:04:30.066569+0000 0'0 2024-09-13T12:04:30.066569+0000 0 0 periodic scrub scheduled @ 2024-09-14T16:54:34.063909+0000 0 0 2024-09-13T12:05:19.501 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:05:19.501 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:05:19.501 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:05:19.515 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-09-13T12:05:19.515 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2024-09-13T12:05:19.515 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-13T12:05:19.516 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:05:19.516 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T12:05:19.516 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:05:19.516 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/0 2024-09-13T12:05:19.516 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T12:05:19.516 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:05:19.516 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:05:19.517 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:05:19.517 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-13T12:05:19.517 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-13T12:05:19.517 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:05:19.517 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:05:19.517 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T12:05:19.517 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:05:19.517 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:05:19.518 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:05:19.518 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:05:19.518 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:05:19.518 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:05:19.518 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:05:19.518 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:05:19.518 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:05:19.519 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T12:05:19.519 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T12:05:19.519 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:05:19.519 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:05:19.519 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:05:19.519 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:05:19.519 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:05:19.520 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:05:19.520 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/0 2024-09-13T12:05:19.520 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.0 2024-09-13T12:05:19.521 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T12:05:19.522 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:05:19.522 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/0/whoami 2024-09-13T12:05:19.522 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T12:05:19.522 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:05:19.523 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:05:19.523 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:05:19.550 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:05:19.549+0000 7f55203b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:05:19.552 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:05:19.551+0000 7f55203b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:05:19.553 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:05:19.552+0000 7f55203b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:05:19.924 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T12:05:19.924 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:05:19.924 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T12:05:19.924 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:05:19.924 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:05:19.925 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:05:19.925 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:05:19.925 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:05:19.925 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:05:19.925 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:05:20.370 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:05:20.912 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:05:20.911+0000 7f55203b2640 -1 Falling back to public interface 2024-09-13T12:05:21.372 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:05:21.372 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:05:21.372 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:05:21.373 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:05:21.373 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:05:21.373 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:05:21.830 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:05:22.831 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:05:22.831 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:05:22.832 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:05:22.832 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:05:22.832 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:05:22.832 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:05:23.251 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:05:24.216 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:05:24.215+0000 7f55203b2640 -1 osd.0 29 log_to_monitors true 2024-09-13T12:05:24.252 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:05:24.252 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:05:24.252 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:05:24.253 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:05:24.253 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:05:24.253 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:05:24.738 INFO:tasks.workunit.client.0.smithi032.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/3817518293,v1:127.0.0.1:6803/3817518293] [v2:127.0.0.1:6804/3817518293,v1:127.0.0.1:6805/3817518293] exists,up 940cd0ac-d018-458e-a8bb-6eb19147770c 2024-09-13T12:05:24.741 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:05:24.741 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:05:24.741 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:05:24.742 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-09-13T12:05:24.742 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2024-09-13T12:05:24.742 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-13T12:05:24.742 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:05:24.742 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-13T12:05:24.742 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:05:24.742 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/2 2024-09-13T12:05:24.742 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T12:05:24.742 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:05:24.742 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:05:24.742 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:05:24.742 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-13T12:05:24.742 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-13T12:05:24.743 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:05:24.743 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:05:24.743 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T12:05:24.743 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:05:24.743 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:05:24.743 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:05:24.743 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:05:24.743 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:05:24.743 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:05:24.743 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:05:24.743 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:05:24.743 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:05:24.743 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T12:05:24.743 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T12:05:24.744 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:05:24.744 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:05:24.744 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:05:24.744 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:05:24.744 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:05:24.744 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:05:24.744 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/2 2024-09-13T12:05:24.744 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.2 2024-09-13T12:05:24.745 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-13T12:05:24.745 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:05:24.745 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/2/whoami 2024-09-13T12:05:24.745 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-13T12:05:24.745 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:05:24.746 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:05:24.746 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:05:24.777 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:05:24.776+0000 7f528b7cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:05:24.784 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:05:24.783+0000 7f528b7cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:05:24.785 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:05:24.784+0000 7f528b7cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:05:25.281 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-13T12:05:25.281 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:05:25.281 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T12:05:25.281 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:05:25.281 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:05:25.281 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:05:25.281 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:05:25.282 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:05:25.282 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:05:25.282 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:05:25.732 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:05:25.899 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:05:25.898+0000 7f528b7cc640 -1 Falling back to public interface 2024-09-13T12:05:26.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:05:26.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:05:26.734 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:05:26.735 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:05:26.735 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:05:26.735 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:05:27.258 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:05:28.260 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:05:28.260 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:05:28.260 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:05:28.260 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:05:28.261 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:05:28.261 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:05:28.679 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:05:28.821 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:05:28.820+0000 7f528b7cc640 -1 osd.2 29 log_to_monitors true 2024-09-13T12:05:29.681 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:05:29.681 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:05:29.681 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:05:29.682 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:05:29.682 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:05:29.682 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:05:30.129 INFO:tasks.workunit.client.0.smithi032.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/746434650,v1:127.0.0.1:6811/746434650] [v2:127.0.0.1:6812/746434650,v1:127.0.0.1:6813/746434650] exists,up 69901422-110f-4bc1-8cb2-72564b47a69b 2024-09-13T12:05:30.130 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:05:30.130 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:05:30.130 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:05:30.130 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-09-13T12:05:30.130 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2024-09-13T12:05:30.694 INFO:tasks.workunit.client.0.smithi032.stderr:osd.0 is already in. 2024-09-13T12:05:30.706 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-09-13T12:05:30.706 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2024-09-13T12:05:31.248 INFO:tasks.workunit.client.0.smithi032.stderr:osd.2 is already in. 2024-09-13T12:05:31.260 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-13T12:05:31.260 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2024-09-13T12:05:31.260 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T12:05:31.260 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T12:05:31.260 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T12:05:31.260 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-13T12:05:31.261 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T12:05:31.261 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T12:05:31.261 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T12:05:31.261 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T12:05:31.261 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T12:05:31.316 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-13T12:05:31.317 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T12:05:31.317 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T12:05:31.317 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T12:05:31.317 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T12:05:31.317 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T12:05:31.830 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T12:05:31.830 INFO:tasks.workunit.client.0.smithi032.stderr:1 2024-09-13T12:05:31.830 INFO:tasks.workunit.client.0.smithi032.stderr:2' 2024-09-13T12:05:31.830 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T12:05:31.830 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:05:31.830 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T12:05:31.985 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822660 2024-09-13T12:05:31.986 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822660 2024-09-13T12:05:31.986 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822660' 2024-09-13T12:05:31.986 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:05:31.986 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T12:05:32.095 INFO:tasks.workunit.client.0.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-13T12:05:32.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-13T12:05:32.097 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-13T12:05:32.097 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-13T12:05:32.097 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:05:32.097 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T12:05:32.221 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757251 2024-09-13T12:05:32.221 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757251 2024-09-13T12:05:32.221 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822660 2-163208757251' 2024-09-13T12:05:32.221 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:05:32.222 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-154618822660 2024-09-13T12:05:32.222 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:05:32.222 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T12:05:32.223 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-154618822660 2024-09-13T12:05:32.223 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:05:32.224 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822660 2024-09-13T12:05:32.224 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2024-09-13T12:05:32.224 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.0 seq 154618822660 2024-09-13T12:05:32.225 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:05:32.662 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822658 -lt 154618822660 2024-09-13T12:05:32.662 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T12:05:33.663 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T12:05:33.664 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:05:34.123 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822660 -lt 154618822660 2024-09-13T12:05:34.123 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:05:34.124 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-163208757251 2024-09-13T12:05:34.124 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:05:34.125 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T12:05:34.125 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-163208757251 2024-09-13T12:05:34.125 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:05:34.126 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757251 2024-09-13T12:05:34.126 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.2 seq 163208757251 2024-09-13T12:05:34.127 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2024-09-13T12:05:34.127 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T12:05:34.561 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757251 -lt 163208757251 2024-09-13T12:05:34.561 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T12:05:34.561 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:05:34.561 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:05:35.109 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T12:05:35.109 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:05:35.109 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:05:35.109 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:05:35.109 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:05:35.109 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:05:35.111 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:05:35.111 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:05:35.547 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:05:35.548 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:05:35.548 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:05:35.548 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:05:36.105 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:05:36.106 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-13T12:05:36.106 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T12:05:36.106 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-13T12:05:36.106 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:05:36.106 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T12:05:36.205 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:05:36.205 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:05:36.205 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:05:36.205 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:05:36.205 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:05:36.205 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:05:36.206 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:05:36.206 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:05:36.603 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:05:36.603 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:05:36.604 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:05:36.604 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:05:37.157 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:05:37.157 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:05:37.157 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:05:37.157 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:05:37.157 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:05:37.157 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:05:37.157 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:05:37.157 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:05:37.157 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:05:37.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:05:37.719 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:05:37.719 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-13T12:05:37.719 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:05:37.719 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T12:05:37.920 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:05:37.920 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:05:37.921 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:05:37.921 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:05:37.921 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:05:37.921 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:05:37.921 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:05:37.921 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:05:38.322 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:05:38.323 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:05:38.323 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:05:38.323 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:05:38.883 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:05:38.883 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:05:38.883 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:05:38.883 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:05:38.883 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:05:38.883 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:05:38.883 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:05:38.884 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:05:38.884 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:05:39.458 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:05:39.458 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:05:39.458 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-13T12:05:39.458 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:05:39.458 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-13T12:05:39.860 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:05:39.860 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:05:39.860 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:05:39.860 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:05:39.860 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:05:39.860 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:05:39.860 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:05:39.860 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:05:40.265 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:05:40.265 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:05:40.266 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:05:40.266 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:05:40.830 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:05:40.830 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:05:40.830 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:05:40.830 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:05:40.831 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:05:40.831 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:05:40.831 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:05:40.831 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:05:40.831 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:05:41.386 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:05:41.386 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:05:41.386 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-13T12:05:41.386 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:05:41.386 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-13T12:05:42.187 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:05:42.187 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:05:42.187 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:05:42.188 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:05:42.188 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:05:42.188 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:05:42.188 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:05:42.188 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:05:42.608 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:05:42.608 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:05:42.609 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:05:42.609 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:05:43.168 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:05:43.168 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:05:43.169 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:05:43.169 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:05:43.169 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:05:43.169 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:05:43.169 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:05:43.169 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:05:43.169 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:05:43.731 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:05:43.731 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:05:43.731 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-13T12:05:43.731 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:05:43.731 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-13T12:05:45.332 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:05:45.332 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:05:45.333 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:05:45.333 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:05:45.333 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:05:45.333 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:05:45.333 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:05:45.333 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:05:45.756 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:05:45.756 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:05:45.756 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:05:45.756 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:05:46.325 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:05:46.326 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:05:46.326 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:05:46.326 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:05:46.326 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:05:46.326 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:05:46.326 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:05:46.326 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:05:46.326 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:05:46.885 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:05:46.885 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:05:46.885 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-13T12:05:46.885 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:05:46.885 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-13T12:05:50.085 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:05:50.086 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:05:50.086 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:05:50.087 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:05:50.087 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:05:50.087 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:05:50.087 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:05:50.087 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:05:50.498 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:05:50.498 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:05:50.498 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:05:50.499 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:05:51.054 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:05:51.054 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:05:51.054 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:05:51.054 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:05:51.054 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:05:51.054 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:05:51.054 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:05:51.054 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:05:51.054 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:05:51.619 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:05:51.620 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:05:51.620 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-13T12:05:51.620 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:05:51.620 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-13T12:05:58.021 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:05:58.021 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:05:58.022 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:05:58.022 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:05:58.022 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:05:58.022 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:05:58.022 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:05:58.022 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:05:58.442 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:05:58.442 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:05:58.443 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:05:58.443 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:05:58.993 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:05:58.993 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:05:58.993 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:05:58.993 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:05:58.993 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:05:58.993 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:05:58.994 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:05:58.994 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:05:58.994 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:05:59.560 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:05:59.560 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:05:59.560 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-13T12:05:59.560 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:05:59.560 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-13T12:06:06.861 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:06:06.862 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:06:06.862 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:06:06.862 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:06:06.863 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:06:06.863 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:06:06.863 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:06:06.863 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:06:07.274 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:06:07.275 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:06:07.275 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:06:07.275 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:06:07.828 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:06:07.828 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:06:07.828 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:06:07.829 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:06:07.829 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:06:07.829 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:06:07.829 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:06:07.829 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:06:07.829 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:06:08.394 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:06:08.394 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:06:08.394 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-13T12:06:08.394 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-13T12:06:08.942 INFO:tasks.workunit.client.0.smithi032.stdout:{ 2024-09-13T12:06:08.942 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_fingerprint": "113f00b8-a374-45c4-82e1-e7adfdcf405d", 2024-09-13T12:06:08.942 INFO:tasks.workunit.client.0.smithi032.stdout: "version": "19.3.0-4697-gd58f7306", 2024-09-13T12:06:08.942 INFO:tasks.workunit.client.0.smithi032.stdout: "commit": "d58f73069a8157a9ace12a0247ee3e8497707ea9", 2024-09-13T12:06:08.942 INFO:tasks.workunit.client.0.smithi032.stdout: "timestamp": "2024-09-13T12:06:08.940445+0000", 2024-09-13T12:06:08.942 INFO:tasks.workunit.client.0.smithi032.stdout: "tag": "", 2024-09-13T12:06:08.942 INFO:tasks.workunit.client.0.smithi032.stdout: "health": { 2024-09-13T12:06:08.943 INFO:tasks.workunit.client.0.smithi032.stdout: "status": "HEALTH_WARN", 2024-09-13T12:06:08.943 INFO:tasks.workunit.client.0.smithi032.stdout: "checks": { 2024-09-13T12:06:08.943 INFO:tasks.workunit.client.0.smithi032.stdout: "OSDMAP_FLAGS": { 2024-09-13T12:06:08.943 INFO:tasks.workunit.client.0.smithi032.stdout: "severity": "HEALTH_WARN", 2024-09-13T12:06:08.943 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": { 2024-09-13T12:06:08.943 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-13T12:06:08.943 INFO:tasks.workunit.client.0.smithi032.stdout: "count": 17 2024-09-13T12:06:08.943 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.943 INFO:tasks.workunit.client.0.smithi032.stdout: "detail": [], 2024-09-13T12:06:08.943 INFO:tasks.workunit.client.0.smithi032.stdout: "muted": false 2024-09-13T12:06:08.944 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.944 INFO:tasks.workunit.client.0.smithi032.stdout: "OSD_DOWN": { 2024-09-13T12:06:08.944 INFO:tasks.workunit.client.0.smithi032.stdout: "severity": "HEALTH_WARN", 2024-09-13T12:06:08.944 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": { 2024-09-13T12:06:08.944 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "1 osds down", 2024-09-13T12:06:08.944 INFO:tasks.workunit.client.0.smithi032.stdout: "count": 1 2024-09-13T12:06:08.944 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.944 INFO:tasks.workunit.client.0.smithi032.stdout: "detail": [ 2024-09-13T12:06:08.944 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.944 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "osd.1 (root=default,host=smithi032) is down" 2024-09-13T12:06:08.944 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:08.944 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:08.945 INFO:tasks.workunit.client.0.smithi032.stdout: "muted": false 2024-09-13T12:06:08.945 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.945 INFO:tasks.workunit.client.0.smithi032.stdout: "PG_DEGRADED": { 2024-09-13T12:06:08.945 INFO:tasks.workunit.client.0.smithi032.stdout: "severity": "HEALTH_WARN", 2024-09-13T12:06:08.945 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": { 2024-09-13T12:06:08.945 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-09-13T12:06:08.945 INFO:tasks.workunit.client.0.smithi032.stdout: "count": 1 2024-09-13T12:06:08.945 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.945 INFO:tasks.workunit.client.0.smithi032.stdout: "detail": [ 2024-09-13T12:06:08.945 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.945 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-09-13T12:06:08.945 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:08.946 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:08.946 INFO:tasks.workunit.client.0.smithi032.stdout: "muted": false 2024-09-13T12:06:08.946 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.946 INFO:tasks.workunit.client.0.smithi032.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-13T12:06:08.946 INFO:tasks.workunit.client.0.smithi032.stdout: "severity": "HEALTH_WARN", 2024-09-13T12:06:08.946 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": { 2024-09-13T12:06:08.946 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-13T12:06:08.946 INFO:tasks.workunit.client.0.smithi032.stdout: "count": 1 2024-09-13T12:06:08.946 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.946 INFO:tasks.workunit.client.0.smithi032.stdout: "detail": [ 2024-09-13T12:06:08.946 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.946 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "application not enabled on pool 'test'" 2024-09-13T12:06:08.946 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.946 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.947 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-13T12:06:08.947 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:08.947 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:08.947 INFO:tasks.workunit.client.0.smithi032.stdout: "muted": false 2024-09-13T12:06:08.947 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:08.947 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.947 INFO:tasks.workunit.client.0.smithi032.stdout: "mutes": [] 2024-09-13T12:06:08.947 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.947 INFO:tasks.workunit.client.0.smithi032.stdout: "monmap_first_committed": 1, 2024-09-13T12:06:08.947 INFO:tasks.workunit.client.0.smithi032.stdout: "monmap_last_committed": 1, 2024-09-13T12:06:08.947 INFO:tasks.workunit.client.0.smithi032.stdout: "monmap": { 2024-09-13T12:06:08.948 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 1, 2024-09-13T12:06:08.948 INFO:tasks.workunit.client.0.smithi032.stdout: "fsid": "e12dfbc3-9bf5-45db-a6f0-f25db93ee351", 2024-09-13T12:06:08.948 INFO:tasks.workunit.client.0.smithi032.stdout: "modified": "2024-09-13T12:02:41.228021Z", 2024-09-13T12:06:08.948 INFO:tasks.workunit.client.0.smithi032.stdout: "created": "2024-09-13T12:02:41.228021Z", 2024-09-13T12:06:08.948 INFO:tasks.workunit.client.0.smithi032.stdout: "min_mon_release": 19, 2024-09-13T12:06:08.948 INFO:tasks.workunit.client.0.smithi032.stdout: "min_mon_release_name": "squid", 2024-09-13T12:06:08.948 INFO:tasks.workunit.client.0.smithi032.stdout: "election_strategy": 1, 2024-09-13T12:06:08.948 INFO:tasks.workunit.client.0.smithi032.stdout: "disallowed_leaders: ": "", 2024-09-13T12:06:08.948 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_mode": false, 2024-09-13T12:06:08.948 INFO:tasks.workunit.client.0.smithi032.stdout: "tiebreaker_mon": "", 2024-09-13T12:06:08.948 INFO:tasks.workunit.client.0.smithi032.stdout: "removed_ranks: ": "", 2024-09-13T12:06:08.949 INFO:tasks.workunit.client.0.smithi032.stdout: "features": { 2024-09-13T12:06:08.949 INFO:tasks.workunit.client.0.smithi032.stdout: "persistent": [ 2024-09-13T12:06:08.949 INFO:tasks.workunit.client.0.smithi032.stdout: "kraken", 2024-09-13T12:06:08.949 INFO:tasks.workunit.client.0.smithi032.stdout: "luminous", 2024-09-13T12:06:08.949 INFO:tasks.workunit.client.0.smithi032.stdout: "mimic", 2024-09-13T12:06:08.949 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap-prune", 2024-09-13T12:06:08.949 INFO:tasks.workunit.client.0.smithi032.stdout: "nautilus", 2024-09-13T12:06:08.949 INFO:tasks.workunit.client.0.smithi032.stdout: "octopus", 2024-09-13T12:06:08.949 INFO:tasks.workunit.client.0.smithi032.stdout: "pacific", 2024-09-13T12:06:08.949 INFO:tasks.workunit.client.0.smithi032.stdout: "elector-pinging", 2024-09-13T12:06:08.949 INFO:tasks.workunit.client.0.smithi032.stdout: "quincy", 2024-09-13T12:06:08.949 INFO:tasks.workunit.client.0.smithi032.stdout: "reef", 2024-09-13T12:06:08.949 INFO:tasks.workunit.client.0.smithi032.stdout: "squid" 2024-09-13T12:06:08.950 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:08.950 INFO:tasks.workunit.client.0.smithi032.stdout: "optional": [] 2024-09-13T12:06:08.950 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.950 INFO:tasks.workunit.client.0.smithi032.stdout: "mons": [ 2024-09-13T12:06:08.950 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.950 INFO:tasks.workunit.client.0.smithi032.stdout: "rank": 0, 2024-09-13T12:06:08.950 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "a", 2024-09-13T12:06:08.950 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addrs": { 2024-09-13T12:06:08.950 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:06:08.950 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.950 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:06:08.950 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:7115", 2024-09-13T12:06:08.951 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 0 2024-09-13T12:06:08.951 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:08.951 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:08.951 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.951 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:7115/0", 2024-09-13T12:06:08.951 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-13T12:06:08.951 INFO:tasks.workunit.client.0.smithi032.stdout: "priority": 0, 2024-09-13T12:06:08.951 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 0, 2024-09-13T12:06:08.951 INFO:tasks.workunit.client.0.smithi032.stdout: "crush_location": "{}" 2024-09-13T12:06:08.951 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:08.951 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:08.951 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.951 INFO:tasks.workunit.client.0.smithi032.stdout: "quorum": [ 2024-09-13T12:06:08.952 INFO:tasks.workunit.client.0.smithi032.stdout: 0 2024-09-13T12:06:08.952 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:08.952 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap": { 2024-09-13T12:06:08.952 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 41, 2024-09-13T12:06:08.952 INFO:tasks.workunit.client.0.smithi032.stdout: "fsid": "e12dfbc3-9bf5-45db-a6f0-f25db93ee351", 2024-09-13T12:06:08.952 INFO:tasks.workunit.client.0.smithi032.stdout: "created": "2024-09-13T12:02:42.168711+0000", 2024-09-13T12:06:08.952 INFO:tasks.workunit.client.0.smithi032.stdout: "modified": "2024-09-13T12:05:31.594789+0000", 2024-09-13T12:06:08.952 INFO:tasks.workunit.client.0.smithi032.stdout: "last_up_change": "2024-09-13T12:05:29.113086+0000", 2024-09-13T12:06:08.952 INFO:tasks.workunit.client.0.smithi032.stdout: "last_in_change": "2024-09-13T12:05:31.074457+0000", 2024-09-13T12:06:08.953 INFO:tasks.workunit.client.0.smithi032.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T12:06:08.953 INFO:tasks.workunit.client.0.smithi032.stdout: "flags_num": 5800384, 2024-09-13T12:06:08.953 INFO:tasks.workunit.client.0.smithi032.stdout: "flags_set": [ 2024-09-13T12:06:08.953 INFO:tasks.workunit.client.0.smithi032.stdout: "nodown", 2024-09-13T12:06:08.953 INFO:tasks.workunit.client.0.smithi032.stdout: "noin", 2024-09-13T12:06:08.953 INFO:tasks.workunit.client.0.smithi032.stdout: "noout", 2024-09-13T12:06:08.953 INFO:tasks.workunit.client.0.smithi032.stdout: "pglog_hardlimit", 2024-09-13T12:06:08.953 INFO:tasks.workunit.client.0.smithi032.stdout: "purged_snapdirs", 2024-09-13T12:06:08.953 INFO:tasks.workunit.client.0.smithi032.stdout: "recovery_deletes", 2024-09-13T12:06:08.953 INFO:tasks.workunit.client.0.smithi032.stdout: "sortbitwise" 2024-09-13T12:06:08.954 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:08.954 INFO:tasks.workunit.client.0.smithi032.stdout: "crush_version": 9, 2024-09-13T12:06:08.954 INFO:tasks.workunit.client.0.smithi032.stdout: "full_ratio": 0.99000000953674316, 2024-09-13T12:06:08.954 INFO:tasks.workunit.client.0.smithi032.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-13T12:06:08.954 INFO:tasks.workunit.client.0.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T12:06:08.954 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_snapshot": "", 2024-09-13T12:06:08.954 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_max": 1, 2024-09-13T12:06:08.954 INFO:tasks.workunit.client.0.smithi032.stdout: "max_osd": 3, 2024-09-13T12:06:08.954 INFO:tasks.workunit.client.0.smithi032.stdout: "require_min_compat_client": "luminous", 2024-09-13T12:06:08.954 INFO:tasks.workunit.client.0.smithi032.stdout: "min_compat_client": "jewel", 2024-09-13T12:06:08.954 INFO:tasks.workunit.client.0.smithi032.stdout: "require_osd_release": "squid", 2024-09-13T12:06:08.954 INFO:tasks.workunit.client.0.smithi032.stdout: "allow_crimson": false, 2024-09-13T12:06:08.954 INFO:tasks.workunit.client.0.smithi032.stdout: "pools": [ 2024-09-13T12:06:08.955 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.955 INFO:tasks.workunit.client.0.smithi032.stdout: "pool": 1, 2024-09-13T12:06:08.955 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_name": "test", 2024-09-13T12:06:08.955 INFO:tasks.workunit.client.0.smithi032.stdout: "create_time": "2024-09-13T12:04:29.912487+0000", 2024-09-13T12:06:08.955 INFO:tasks.workunit.client.0.smithi032.stdout: "flags": 1, 2024-09-13T12:06:08.955 INFO:tasks.workunit.client.0.smithi032.stdout: "flags_names": "hashpspool", 2024-09-13T12:06:08.955 INFO:tasks.workunit.client.0.smithi032.stdout: "type": 1, 2024-09-13T12:06:08.955 INFO:tasks.workunit.client.0.smithi032.stdout: "size": 3, 2024-09-13T12:06:08.955 INFO:tasks.workunit.client.0.smithi032.stdout: "min_size": 2, 2024-09-13T12:06:08.955 INFO:tasks.workunit.client.0.smithi032.stdout: "crush_rule": 0, 2024-09-13T12:06:08.956 INFO:tasks.workunit.client.0.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-13T12:06:08.956 INFO:tasks.workunit.client.0.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-13T12:06:08.956 INFO:tasks.workunit.client.0.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T12:06:08.956 INFO:tasks.workunit.client.0.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T12:06:08.956 INFO:tasks.workunit.client.0.smithi032.stdout: "is_stretch_pool": false, 2024-09-13T12:06:08.956 INFO:tasks.workunit.client.0.smithi032.stdout: "object_hash": 2, 2024-09-13T12:06:08.956 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_autoscale_mode": "off", 2024-09-13T12:06:08.956 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_num": 1, 2024-09-13T12:06:08.956 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_placement_num": 1, 2024-09-13T12:06:08.956 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_placement_num_target": 1, 2024-09-13T12:06:08.956 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_num_target": 1, 2024-09-13T12:06:08.956 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_num_pending": 1, 2024-09-13T12:06:08.957 INFO:tasks.workunit.client.0.smithi032.stdout: "last_pg_merge_meta": { 2024-09-13T12:06:08.957 INFO:tasks.workunit.client.0.smithi032.stdout: "source_pgid": "0.0", 2024-09-13T12:06:08.957 INFO:tasks.workunit.client.0.smithi032.stdout: "ready_epoch": 0, 2024-09-13T12:06:08.957 INFO:tasks.workunit.client.0.smithi032.stdout: "last_epoch_started": 0, 2024-09-13T12:06:08.957 INFO:tasks.workunit.client.0.smithi032.stdout: "last_epoch_clean": 0, 2024-09-13T12:06:08.957 INFO:tasks.workunit.client.0.smithi032.stdout: "source_version": "0'0", 2024-09-13T12:06:08.957 INFO:tasks.workunit.client.0.smithi032.stdout: "target_version": "0'0" 2024-09-13T12:06:08.957 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.957 INFO:tasks.workunit.client.0.smithi032.stdout: "last_change": "29", 2024-09-13T12:06:08.957 INFO:tasks.workunit.client.0.smithi032.stdout: "last_force_op_resend": "0", 2024-09-13T12:06:08.957 INFO:tasks.workunit.client.0.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T12:06:08.958 INFO:tasks.workunit.client.0.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T12:06:08.958 INFO:tasks.workunit.client.0.smithi032.stdout: "auid": 0, 2024-09-13T12:06:08.958 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_mode": "selfmanaged", 2024-09-13T12:06:08.958 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_seq": 0, 2024-09-13T12:06:08.958 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_epoch": 0, 2024-09-13T12:06:08.958 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_snaps": [], 2024-09-13T12:06:08.958 INFO:tasks.workunit.client.0.smithi032.stdout: "removed_snaps": "[]", 2024-09-13T12:06:08.958 INFO:tasks.workunit.client.0.smithi032.stdout: "quota_max_bytes": 0, 2024-09-13T12:06:08.958 INFO:tasks.workunit.client.0.smithi032.stdout: "quota_max_objects": 0, 2024-09-13T12:06:08.958 INFO:tasks.workunit.client.0.smithi032.stdout: "tiers": [], 2024-09-13T12:06:08.958 INFO:tasks.workunit.client.0.smithi032.stdout: "tier_of": -1, 2024-09-13T12:06:08.958 INFO:tasks.workunit.client.0.smithi032.stdout: "read_tier": -1, 2024-09-13T12:06:08.959 INFO:tasks.workunit.client.0.smithi032.stdout: "write_tier": -1, 2024-09-13T12:06:08.959 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_mode": "none", 2024-09-13T12:06:08.959 INFO:tasks.workunit.client.0.smithi032.stdout: "target_max_bytes": 0, 2024-09-13T12:06:08.959 INFO:tasks.workunit.client.0.smithi032.stdout: "target_max_objects": 0, 2024-09-13T12:06:08.959 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T12:06:08.959 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T12:06:08.959 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T12:06:08.959 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-13T12:06:08.959 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-13T12:06:08.960 INFO:tasks.workunit.client.0.smithi032.stdout: "erasure_code_profile": "", 2024-09-13T12:06:08.960 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_params": { 2024-09-13T12:06:08.960 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "none" 2024-09-13T12:06:08.960 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.960 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_period": 0, 2024-09-13T12:06:08.960 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_count": 0, 2024-09-13T12:06:08.960 INFO:tasks.workunit.client.0.smithi032.stdout: "use_gmt_hitset": true, 2024-09-13T12:06:08.960 INFO:tasks.workunit.client.0.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-13T12:06:08.960 INFO:tasks.workunit.client.0.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-13T12:06:08.960 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T12:06:08.960 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-13T12:06:08.960 INFO:tasks.workunit.client.0.smithi032.stdout: "grade_table": [], 2024-09-13T12:06:08.961 INFO:tasks.workunit.client.0.smithi032.stdout: "stripe_width": 0, 2024-09-13T12:06:08.961 INFO:tasks.workunit.client.0.smithi032.stdout: "expected_num_objects": 0, 2024-09-13T12:06:08.961 INFO:tasks.workunit.client.0.smithi032.stdout: "fast_read": false, 2024-09-13T12:06:08.961 INFO:tasks.workunit.client.0.smithi032.stdout: "options": {}, 2024-09-13T12:06:08.961 INFO:tasks.workunit.client.0.smithi032.stdout: "application_metadata": {}, 2024-09-13T12:06:08.961 INFO:tasks.workunit.client.0.smithi032.stdout: "read_balance": { 2024-09-13T12:06:08.961 INFO:tasks.workunit.client.0.smithi032.stdout: "score_type": "Fair distribution", 2024-09-13T12:06:08.961 INFO:tasks.workunit.client.0.smithi032.stdout: "score_acting": 2.9900000095367432, 2024-09-13T12:06:08.961 INFO:tasks.workunit.client.0.smithi032.stdout: "score_stable": 2.9900000095367432, 2024-09-13T12:06:08.962 INFO:tasks.workunit.client.0.smithi032.stdout: "optimal_score": 0.67000001668930054, 2024-09-13T12:06:08.962 INFO:tasks.workunit.client.0.smithi032.stdout: "raw_score_acting": 2, 2024-09-13T12:06:08.962 INFO:tasks.workunit.client.0.smithi032.stdout: "raw_score_stable": 2, 2024-09-13T12:06:08.962 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_affinity_weighted": 1, 2024-09-13T12:06:08.962 INFO:tasks.workunit.client.0.smithi032.stdout: "average_primary_affinity": 1, 2024-09-13T12:06:08.962 INFO:tasks.workunit.client.0.smithi032.stdout: "average_primary_affinity_weighted": 1 2024-09-13T12:06:08.962 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:08.962 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:08.962 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:08.962 INFO:tasks.workunit.client.0.smithi032.stdout: "osds": [ 2024-09-13T12:06:08.962 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.962 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 0, 2024-09-13T12:06:08.962 INFO:tasks.workunit.client.0.smithi032.stdout: "uuid": "940cd0ac-d018-458e-a8bb-6eb19147770c", 2024-09-13T12:06:08.963 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 1, 2024-09-13T12:06:08.963 INFO:tasks.workunit.client.0.smithi032.stdout: "in": 1, 2024-09-13T12:06:08.963 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 1, 2024-09-13T12:06:08.963 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_affinity": 1, 2024-09-13T12:06:08.963 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_begin": 5, 2024-09-13T12:06:08.963 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_end": 29, 2024-09-13T12:06:08.963 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 36, 2024-09-13T12:06:08.963 INFO:tasks.workunit.client.0.smithi032.stdout: "up_thru": 40, 2024-09-13T12:06:08.963 INFO:tasks.workunit.client.0.smithi032.stdout: "down_at": 30, 2024-09-13T12:06:08.963 INFO:tasks.workunit.client.0.smithi032.stdout: "lost_at": 0, 2024-09-13T12:06:08.963 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addrs": { 2024-09-13T12:06:08.963 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:06:08.964 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.964 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:06:08.964 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6802", 2024-09-13T12:06:08.964 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3817518293 2024-09-13T12:06:08.964 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.964 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.964 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:06:08.964 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6803", 2024-09-13T12:06:08.964 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3817518293 2024-09-13T12:06:08.964 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:08.964 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:08.964 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.964 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addrs": { 2024-09-13T12:06:08.965 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:06:08.965 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.965 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:06:08.965 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6804", 2024-09-13T12:06:08.965 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3817518293 2024-09-13T12:06:08.965 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.965 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.965 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:06:08.965 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6805", 2024-09-13T12:06:08.965 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3817518293 2024-09-13T12:06:08.966 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:08.966 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:08.966 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.966 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-13T12:06:08.966 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:06:08.966 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.966 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:06:08.966 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6808", 2024-09-13T12:06:08.966 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3817518293 2024-09-13T12:06:08.966 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.966 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.966 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:06:08.967 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6809", 2024-09-13T12:06:08.967 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3817518293 2024-09-13T12:06:08.967 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:08.967 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:08.967 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.967 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-13T12:06:08.967 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:06:08.967 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.967 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:06:08.967 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6806", 2024-09-13T12:06:08.967 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3817518293 2024-09-13T12:06:08.967 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.968 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.968 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:06:08.968 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6807", 2024-09-13T12:06:08.968 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3817518293 2024-09-13T12:06:08.968 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:08.968 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:08.968 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.968 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addr": "127.0.0.1:6803/3817518293", 2024-09-13T12:06:08.968 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addr": "127.0.0.1:6805/3817518293", 2024-09-13T12:06:08.968 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addr": "127.0.0.1:6809/3817518293", 2024-09-13T12:06:08.968 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addr": "127.0.0.1:6807/3817518293", 2024-09-13T12:06:08.969 INFO:tasks.workunit.client.0.smithi032.stdout: "state": [ 2024-09-13T12:06:08.969 INFO:tasks.workunit.client.0.smithi032.stdout: "exists", 2024-09-13T12:06:08.969 INFO:tasks.workunit.client.0.smithi032.stdout: "up" 2024-09-13T12:06:08.969 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:08.969 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.969 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.969 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 1, 2024-09-13T12:06:08.969 INFO:tasks.workunit.client.0.smithi032.stdout: "uuid": "eb913216-75c2-4cc2-ad3b-855fbbad0fc3", 2024-09-13T12:06:08.969 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 0, 2024-09-13T12:06:08.969 INFO:tasks.workunit.client.0.smithi032.stdout: "in": 1, 2024-09-13T12:06:08.969 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 1, 2024-09-13T12:06:08.970 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_affinity": 1, 2024-09-13T12:06:08.970 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_begin": 0, 2024-09-13T12:06:08.970 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_end": 0, 2024-09-13T12:06:08.970 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 10, 2024-09-13T12:06:08.970 INFO:tasks.workunit.client.0.smithi032.stdout: "up_thru": 23, 2024-09-13T12:06:08.970 INFO:tasks.workunit.client.0.smithi032.stdout: "down_at": 31, 2024-09-13T12:06:08.970 INFO:tasks.workunit.client.0.smithi032.stdout: "lost_at": 0, 2024-09-13T12:06:08.970 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addrs": { 2024-09-13T12:06:08.970 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:06:08.970 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.970 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:06:08.970 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6810", 2024-09-13T12:06:08.970 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1355014695 2024-09-13T12:06:08.971 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.971 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.971 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:06:08.971 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6811", 2024-09-13T12:06:08.971 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1355014695 2024-09-13T12:06:08.971 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:08.971 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:08.971 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.971 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addrs": { 2024-09-13T12:06:08.971 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:06:08.971 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.971 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:06:08.972 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6812", 2024-09-13T12:06:08.972 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1355014695 2024-09-13T12:06:08.972 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.972 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.972 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:06:08.972 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6813", 2024-09-13T12:06:08.972 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1355014695 2024-09-13T12:06:08.972 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:08.972 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:08.972 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.972 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-13T12:06:08.972 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:06:08.972 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.973 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:06:08.973 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6816", 2024-09-13T12:06:08.973 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1355014695 2024-09-13T12:06:08.973 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.973 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.973 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:06:08.973 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6817", 2024-09-13T12:06:08.973 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1355014695 2024-09-13T12:06:08.973 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:08.973 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:08.973 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.973 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-13T12:06:08.973 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:06:08.974 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.974 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:06:08.974 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6814", 2024-09-13T12:06:08.974 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1355014695 2024-09-13T12:06:08.974 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.974 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.974 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:06:08.974 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6815", 2024-09-13T12:06:08.974 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1355014695 2024-09-13T12:06:08.974 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:08.974 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:08.974 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.974 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addr": "127.0.0.1:6811/1355014695", 2024-09-13T12:06:08.975 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addr": "127.0.0.1:6813/1355014695", 2024-09-13T12:06:08.975 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1355014695", 2024-09-13T12:06:08.975 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1355014695", 2024-09-13T12:06:08.975 INFO:tasks.workunit.client.0.smithi032.stdout: "state": [ 2024-09-13T12:06:08.975 INFO:tasks.workunit.client.0.smithi032.stdout: "exists" 2024-09-13T12:06:08.975 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:08.975 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.975 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.975 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2, 2024-09-13T12:06:08.975 INFO:tasks.workunit.client.0.smithi032.stdout: "uuid": "69901422-110f-4bc1-8cb2-72564b47a69b", 2024-09-13T12:06:08.975 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 1, 2024-09-13T12:06:08.975 INFO:tasks.workunit.client.0.smithi032.stdout: "in": 1, 2024-09-13T12:06:08.975 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 1, 2024-09-13T12:06:08.976 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_affinity": 1, 2024-09-13T12:06:08.976 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_begin": 15, 2024-09-13T12:06:08.976 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_end": 29, 2024-09-13T12:06:08.976 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 38, 2024-09-13T12:06:08.976 INFO:tasks.workunit.client.0.smithi032.stdout: "up_thru": 0, 2024-09-13T12:06:08.976 INFO:tasks.workunit.client.0.smithi032.stdout: "down_at": 32, 2024-09-13T12:06:08.976 INFO:tasks.workunit.client.0.smithi032.stdout: "lost_at": 0, 2024-09-13T12:06:08.976 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addrs": { 2024-09-13T12:06:08.977 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:06:08.977 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.977 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:06:08.977 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6810", 2024-09-13T12:06:08.977 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 746434650 2024-09-13T12:06:08.977 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.977 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.977 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:06:08.977 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6811", 2024-09-13T12:06:08.977 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 746434650 2024-09-13T12:06:08.977 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:08.977 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:08.978 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.978 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addrs": { 2024-09-13T12:06:08.978 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:06:08.978 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.978 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:06:08.978 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6812", 2024-09-13T12:06:08.978 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 746434650 2024-09-13T12:06:08.978 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.978 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.979 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:06:08.979 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6813", 2024-09-13T12:06:08.979 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 746434650 2024-09-13T12:06:08.979 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:08.979 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:08.979 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.979 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-13T12:06:08.979 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:06:08.979 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.979 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:06:08.979 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6816", 2024-09-13T12:06:08.980 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 746434650 2024-09-13T12:06:08.980 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.980 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.982 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:06:08.982 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6817", 2024-09-13T12:06:08.982 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 746434650 2024-09-13T12:06:08.982 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:08.982 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:08.982 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.982 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-13T12:06:08.983 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:06:08.983 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.983 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:06:08.983 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6814", 2024-09-13T12:06:08.983 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 746434650 2024-09-13T12:06:08.983 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.983 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.983 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:06:08.983 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6815", 2024-09-13T12:06:08.984 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 746434650 2024-09-13T12:06:08.984 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:08.984 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:08.984 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.984 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addr": "127.0.0.1:6811/746434650", 2024-09-13T12:06:08.984 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addr": "127.0.0.1:6813/746434650", 2024-09-13T12:06:08.984 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addr": "127.0.0.1:6817/746434650", 2024-09-13T12:06:08.984 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addr": "127.0.0.1:6815/746434650", 2024-09-13T12:06:08.985 INFO:tasks.workunit.client.0.smithi032.stdout: "state": [ 2024-09-13T12:06:08.985 INFO:tasks.workunit.client.0.smithi032.stdout: "exists", 2024-09-13T12:06:08.985 INFO:tasks.workunit.client.0.smithi032.stdout: "up" 2024-09-13T12:06:08.985 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:08.986 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:08.986 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:08.986 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_xinfo": [ 2024-09-13T12:06:08.986 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.986 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 0, 2024-09-13T12:06:08.986 INFO:tasks.workunit.client.0.smithi032.stdout: "down_stamp": "2024-09-13T12:05:16.553008+0000", 2024-09-13T12:06:08.986 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_probability": 0, 2024-09-13T12:06:08.986 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_interval": 0, 2024-09-13T12:06:08.986 INFO:tasks.workunit.client.0.smithi032.stdout: "features": 4540719139924082687, 2024-09-13T12:06:08.986 INFO:tasks.workunit.client.0.smithi032.stdout: "old_weight": 0, 2024-09-13T12:06:08.986 INFO:tasks.workunit.client.0.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-13T12:03:15.350153+0000", 2024-09-13T12:06:08.986 INFO:tasks.workunit.client.0.smithi032.stdout: "dead_epoch": 0 2024-09-13T12:06:08.986 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.987 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.987 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 1, 2024-09-13T12:06:08.987 INFO:tasks.workunit.client.0.smithi032.stdout: "down_stamp": "2024-09-13T12:05:17.080406+0000", 2024-09-13T12:06:08.987 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_probability": 0, 2024-09-13T12:06:08.987 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_interval": 0, 2024-09-13T12:06:08.987 INFO:tasks.workunit.client.0.smithi032.stdout: "features": 4540719139924082687, 2024-09-13T12:06:08.987 INFO:tasks.workunit.client.0.smithi032.stdout: "old_weight": 0, 2024-09-13T12:06:08.987 INFO:tasks.workunit.client.0.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-13T12:03:51.563515+0000", 2024-09-13T12:06:08.987 INFO:tasks.workunit.client.0.smithi032.stdout: "dead_epoch": 0 2024-09-13T12:06:08.987 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.987 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.987 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2, 2024-09-13T12:06:08.987 INFO:tasks.workunit.client.0.smithi032.stdout: "down_stamp": "2024-09-13T12:05:17.687936+0000", 2024-09-13T12:06:08.988 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_probability": 0, 2024-09-13T12:06:08.988 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_interval": 0, 2024-09-13T12:06:08.988 INFO:tasks.workunit.client.0.smithi032.stdout: "features": 4540719139924082687, 2024-09-13T12:06:08.988 INFO:tasks.workunit.client.0.smithi032.stdout: "old_weight": 0, 2024-09-13T12:06:08.988 INFO:tasks.workunit.client.0.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-13T12:04:26.609710+0000", 2024-09-13T12:06:08.988 INFO:tasks.workunit.client.0.smithi032.stdout: "dead_epoch": 0 2024-09-13T12:06:08.988 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:08.988 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:08.988 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_upmap": [], 2024-09-13T12:06:08.988 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_upmap_items": [], 2024-09-13T12:06:08.988 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_upmap_primaries": [], 2024-09-13T12:06:08.988 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_temp": [], 2024-09-13T12:06:08.988 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_temp": [], 2024-09-13T12:06:08.988 INFO:tasks.workunit.client.0.smithi032.stdout: "blocklist": {}, 2024-09-13T12:06:08.989 INFO:tasks.workunit.client.0.smithi032.stdout: "range_blocklist": {}, 2024-09-13T12:06:08.989 INFO:tasks.workunit.client.0.smithi032.stdout: "erasure_code_profiles": { 2024-09-13T12:06:08.989 INFO:tasks.workunit.client.0.smithi032.stdout: "default": { 2024-09-13T12:06:08.989 INFO:tasks.workunit.client.0.smithi032.stdout: "k": "2", 2024-09-13T12:06:08.989 INFO:tasks.workunit.client.0.smithi032.stdout: "m": "2", 2024-09-13T12:06:08.989 INFO:tasks.workunit.client.0.smithi032.stdout: "plugin": "jerasure", 2024-09-13T12:06:08.989 INFO:tasks.workunit.client.0.smithi032.stdout: "technique": "reed_sol_van" 2024-09-13T12:06:08.989 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:08.989 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.989 INFO:tasks.workunit.client.0.smithi032.stdout: "removed_snaps_queue": [], 2024-09-13T12:06:08.989 INFO:tasks.workunit.client.0.smithi032.stdout: "new_removed_snaps": [], 2024-09-13T12:06:08.989 INFO:tasks.workunit.client.0.smithi032.stdout: "new_purged_snaps": [], 2024-09-13T12:06:08.989 INFO:tasks.workunit.client.0.smithi032.stdout: "crush_node_flags": {}, 2024-09-13T12:06:08.992 INFO:tasks.workunit.client.0.smithi032.stdout: "device_class_flags": {}, 2024-09-13T12:06:08.992 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_mode": { 2024-09-13T12:06:08.992 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_mode_enabled": false, 2024-09-13T12:06:08.992 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_bucket_count": 0, 2024-09-13T12:06:08.992 INFO:tasks.workunit.client.0.smithi032.stdout: "degraded_stretch_mode": 0, 2024-09-13T12:06:08.992 INFO:tasks.workunit.client.0.smithi032.stdout: "recovering_stretch_mode": 0, 2024-09-13T12:06:08.992 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_mode_bucket": 0 2024-09-13T12:06:08.992 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:08.992 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.993 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_metadata": [ 2024-09-13T12:06:08.993 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.993 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:06:08.993 INFO:tasks.workunit.client.0.smithi032.stdout: "arch": "x86_64", 2024-09-13T12:06:08.993 INFO:tasks.workunit.client.0.smithi032.stdout: "back_addr": "[v2:127.0.0.1:6804/3817518293,v1:127.0.0.1:6805/3817518293]", 2024-09-13T12:06:08.993 INFO:tasks.workunit.client.0.smithi032.stdout: "back_iface": "", 2024-09-13T12:06:08.993 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs": "1", 2024-09-13T12:06:08.993 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_db": "0", 2024-09-13T12:06:08.993 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_wal": "0", 2024-09-13T12:06:08.993 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_single_shared_device": "1", 2024-09-13T12:06:08.994 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_allocation_from_file": "0", 2024-09-13T12:06:08.994 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_access_mode": "file", 2024-09-13T12:06:08.994 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_block_size": "4096", 2024-09-13T12:06:08.994 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_devices": "sda", 2024-09-13T12:06:08.994 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-13T12:06:08.994 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-13T12:06:08.994 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-13T12:06:08.994 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_rotational": "1", 2024-09-13T12:06:08.994 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_size": "107374182400", 2024-09-13T12:06:08.994 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_support_discard": "1", 2024-09-13T12:06:08.994 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_type": "hdd", 2024-09-13T12:06:08.994 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_min_alloc_size": "4096", 2024-09-13T12:06:08.995 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_release": "squid", 2024-09-13T12:06:08.995 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:06:08.995 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_short": "19.3.0-4697-gd58f7306", 2024-09-13T12:06:08.995 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_when_created": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:06:08.995 INFO:tasks.workunit.client.0.smithi032.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-13T12:06:08.995 INFO:tasks.workunit.client.0.smithi032.stdout: "created_at": "2024-09-13T12:02:48.448877Z", 2024-09-13T12:06:08.995 INFO:tasks.workunit.client.0.smithi032.stdout: "default_device_class": "hdd", 2024-09-13T12:06:08.995 INFO:tasks.workunit.client.0.smithi032.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4J2LL", 2024-09-13T12:06:08.995 INFO:tasks.workunit.client.0.smithi032.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-13T12:06:08.995 INFO:tasks.workunit.client.0.smithi032.stdout: "devices": "sda", 2024-09-13T12:06:08.995 INFO:tasks.workunit.client.0.smithi032.stdout: "distro": "centos", 2024-09-13T12:06:08.995 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_description": "CentOS Stream 9", 2024-09-13T12:06:08.995 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_version": "9", 2024-09-13T12:06:08.996 INFO:tasks.workunit.client.0.smithi032.stdout: "front_addr": "[v2:127.0.0.1:6802/3817518293,v1:127.0.0.1:6803/3817518293]", 2024-09-13T12:06:08.996 INFO:tasks.workunit.client.0.smithi032.stdout: "front_iface": "", 2024-09-13T12:06:08.996 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/3817518293,v1:127.0.0.1:6809/3817518293]", 2024-09-13T12:06:08.996 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/3817518293,v1:127.0.0.1:6807/3817518293]", 2024-09-13T12:06:08.996 INFO:tasks.workunit.client.0.smithi032.stdout: "hostname": "smithi032", 2024-09-13T12:06:08.996 INFO:tasks.workunit.client.0.smithi032.stdout: "journal_rotational": "1", 2024-09-13T12:06:08.996 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-13T12:06:08.996 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-13T12:06:08.996 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_swap_kb": "0", 2024-09-13T12:06:08.996 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_total_kb": "32493484", 2024-09-13T12:06:08.996 INFO:tasks.workunit.client.0.smithi032.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-13T12:06:08.996 INFO:tasks.workunit.client.0.smithi032.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-13T12:06:08.997 INFO:tasks.workunit.client.0.smithi032.stdout: "os": "Linux", 2024-09-13T12:06:08.997 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_data": "td/divergent-priors/0", 2024-09-13T12:06:08.997 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_objectstore": "bluestore", 2024-09-13T12:06:08.997 INFO:tasks.workunit.client.0.smithi032.stdout: "osdspec_affinity": "", 2024-09-13T12:06:08.997 INFO:tasks.workunit.client.0.smithi032.stdout: "rotational": "1" 2024-09-13T12:06:08.997 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:08.997 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:08.997 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 1, 2024-09-13T12:06:08.997 INFO:tasks.workunit.client.0.smithi032.stdout: "arch": "x86_64", 2024-09-13T12:06:08.997 INFO:tasks.workunit.client.0.smithi032.stdout: "back_addr": "[v2:127.0.0.1:6812/1355014695,v1:127.0.0.1:6813/1355014695]", 2024-09-13T12:06:08.997 INFO:tasks.workunit.client.0.smithi032.stdout: "back_iface": "", 2024-09-13T12:06:08.997 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs": "1", 2024-09-13T12:06:08.997 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_db": "0", 2024-09-13T12:06:08.997 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_wal": "0", 2024-09-13T12:06:08.998 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_single_shared_device": "1", 2024-09-13T12:06:08.998 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_allocation_from_file": "0", 2024-09-13T12:06:08.998 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_access_mode": "file", 2024-09-13T12:06:08.998 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_block_size": "4096", 2024-09-13T12:06:08.998 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_devices": "sda", 2024-09-13T12:06:08.998 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-13T12:06:08.998 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-13T12:06:08.998 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-13T12:06:08.999 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_rotational": "1", 2024-09-13T12:06:08.999 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_size": "107374182400", 2024-09-13T12:06:08.999 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_support_discard": "1", 2024-09-13T12:06:08.999 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_type": "hdd", 2024-09-13T12:06:08.999 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_min_alloc_size": "4096", 2024-09-13T12:06:08.999 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_release": "squid", 2024-09-13T12:06:08.999 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:06:08.999 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_short": "19.3.0-4697-gd58f7306", 2024-09-13T12:06:08.999 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_when_created": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:06:08.999 INFO:tasks.workunit.client.0.smithi032.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-13T12:06:08.999 INFO:tasks.workunit.client.0.smithi032.stdout: "created_at": "2024-09-13T12:03:23.158100Z", 2024-09-13T12:06:08.999 INFO:tasks.workunit.client.0.smithi032.stdout: "default_device_class": "hdd", 2024-09-13T12:06:08.999 INFO:tasks.workunit.client.0.smithi032.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4J2LL", 2024-09-13T12:06:09.000 INFO:tasks.workunit.client.0.smithi032.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-13T12:06:09.000 INFO:tasks.workunit.client.0.smithi032.stdout: "devices": "sda", 2024-09-13T12:06:09.000 INFO:tasks.workunit.client.0.smithi032.stdout: "distro": "centos", 2024-09-13T12:06:09.000 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_description": "CentOS Stream 9", 2024-09-13T12:06:09.000 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_version": "9", 2024-09-13T12:06:09.000 INFO:tasks.workunit.client.0.smithi032.stdout: "front_addr": "[v2:127.0.0.1:6810/1355014695,v1:127.0.0.1:6811/1355014695]", 2024-09-13T12:06:09.000 INFO:tasks.workunit.client.0.smithi032.stdout: "front_iface": "", 2024-09-13T12:06:09.000 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1355014695,v1:127.0.0.1:6817/1355014695]", 2024-09-13T12:06:09.000 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1355014695,v1:127.0.0.1:6815/1355014695]", 2024-09-13T12:06:09.000 INFO:tasks.workunit.client.0.smithi032.stdout: "hostname": "smithi032", 2024-09-13T12:06:09.000 INFO:tasks.workunit.client.0.smithi032.stdout: "journal_rotational": "1", 2024-09-13T12:06:09.000 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-13T12:06:09.001 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-13T12:06:09.001 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_swap_kb": "0", 2024-09-13T12:06:09.001 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_total_kb": "32493484", 2024-09-13T12:06:09.001 INFO:tasks.workunit.client.0.smithi032.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-13T12:06:09.001 INFO:tasks.workunit.client.0.smithi032.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-13T12:06:09.001 INFO:tasks.workunit.client.0.smithi032.stdout: "os": "Linux", 2024-09-13T12:06:09.001 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_data": "td/divergent-priors/1", 2024-09-13T12:06:09.001 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_objectstore": "bluestore", 2024-09-13T12:06:09.001 INFO:tasks.workunit.client.0.smithi032.stdout: "osdspec_affinity": "", 2024-09-13T12:06:09.001 INFO:tasks.workunit.client.0.smithi032.stdout: "rotational": "1" 2024-09-13T12:06:09.001 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.001 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.001 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:06:09.001 INFO:tasks.workunit.client.0.smithi032.stdout: "arch": "x86_64", 2024-09-13T12:06:09.002 INFO:tasks.workunit.client.0.smithi032.stdout: "back_addr": "[v2:127.0.0.1:6812/746434650,v1:127.0.0.1:6813/746434650]", 2024-09-13T12:06:09.002 INFO:tasks.workunit.client.0.smithi032.stdout: "back_iface": "", 2024-09-13T12:06:09.002 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs": "1", 2024-09-13T12:06:09.002 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_db": "0", 2024-09-13T12:06:09.002 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_wal": "0", 2024-09-13T12:06:09.002 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_single_shared_device": "1", 2024-09-13T12:06:09.002 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_allocation_from_file": "0", 2024-09-13T12:06:09.002 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_access_mode": "file", 2024-09-13T12:06:09.002 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_block_size": "4096", 2024-09-13T12:06:09.002 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_devices": "sda", 2024-09-13T12:06:09.002 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-13T12:06:09.002 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-13T12:06:09.002 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-13T12:06:09.003 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_rotational": "1", 2024-09-13T12:06:09.003 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_size": "107374182400", 2024-09-13T12:06:09.003 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_support_discard": "1", 2024-09-13T12:06:09.003 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_type": "hdd", 2024-09-13T12:06:09.003 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_min_alloc_size": "4096", 2024-09-13T12:06:09.003 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_release": "squid", 2024-09-13T12:06:09.003 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:06:09.003 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_short": "19.3.0-4697-gd58f7306", 2024-09-13T12:06:09.003 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_when_created": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:06:09.003 INFO:tasks.workunit.client.0.smithi032.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-13T12:06:09.003 INFO:tasks.workunit.client.0.smithi032.stdout: "created_at": "2024-09-13T12:03:58.721983Z", 2024-09-13T12:06:09.003 INFO:tasks.workunit.client.0.smithi032.stdout: "default_device_class": "hdd", 2024-09-13T12:06:09.003 INFO:tasks.workunit.client.0.smithi032.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4J2LL", 2024-09-13T12:06:09.004 INFO:tasks.workunit.client.0.smithi032.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-13T12:06:09.004 INFO:tasks.workunit.client.0.smithi032.stdout: "devices": "sda", 2024-09-13T12:06:09.004 INFO:tasks.workunit.client.0.smithi032.stdout: "distro": "centos", 2024-09-13T12:06:09.004 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_description": "CentOS Stream 9", 2024-09-13T12:06:09.004 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_version": "9", 2024-09-13T12:06:09.004 INFO:tasks.workunit.client.0.smithi032.stdout: "front_addr": "[v2:127.0.0.1:6810/746434650,v1:127.0.0.1:6811/746434650]", 2024-09-13T12:06:09.004 INFO:tasks.workunit.client.0.smithi032.stdout: "front_iface": "", 2024-09-13T12:06:09.004 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/746434650,v1:127.0.0.1:6817/746434650]", 2024-09-13T12:06:09.004 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/746434650,v1:127.0.0.1:6815/746434650]", 2024-09-13T12:06:09.004 INFO:tasks.workunit.client.0.smithi032.stdout: "hostname": "smithi032", 2024-09-13T12:06:09.004 INFO:tasks.workunit.client.0.smithi032.stdout: "journal_rotational": "1", 2024-09-13T12:06:09.004 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-13T12:06:09.004 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-13T12:06:09.005 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_swap_kb": "0", 2024-09-13T12:06:09.005 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_total_kb": "32493484", 2024-09-13T12:06:09.005 INFO:tasks.workunit.client.0.smithi032.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-13T12:06:09.005 INFO:tasks.workunit.client.0.smithi032.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-13T12:06:09.005 INFO:tasks.workunit.client.0.smithi032.stdout: "os": "Linux", 2024-09-13T12:06:09.005 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_data": "td/divergent-priors/2", 2024-09-13T12:06:09.005 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_objectstore": "bluestore", 2024-09-13T12:06:09.005 INFO:tasks.workunit.client.0.smithi032.stdout: "osdspec_affinity": "", 2024-09-13T12:06:09.005 INFO:tasks.workunit.client.0.smithi032.stdout: "rotational": "1" 2024-09-13T12:06:09.005 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:09.005 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:09.005 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap_clean_epochs": { 2024-09-13T12:06:09.005 INFO:tasks.workunit.client.0.smithi032.stdout: "min_last_epoch_clean": 0, 2024-09-13T12:06:09.006 INFO:tasks.workunit.client.0.smithi032.stdout: "last_epoch_clean": { 2024-09-13T12:06:09.006 INFO:tasks.workunit.client.0.smithi032.stdout: "per_pool": [] 2024-09-13T12:06:09.006 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.006 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_epochs": [ 2024-09-13T12:06:09.006 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.006 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:06:09.006 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 36 2024-09-13T12:06:09.006 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.006 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.006 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:06:09.006 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 38 2024-09-13T12:06:09.006 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:09.006 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:09.007 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.007 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap_first_committed": 1, 2024-09-13T12:06:09.007 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap_last_committed": 41, 2024-09-13T12:06:09.007 INFO:tasks.workunit.client.0.smithi032.stdout: "crushmap": { 2024-09-13T12:06:09.007 INFO:tasks.workunit.client.0.smithi032.stdout: "devices": [ 2024-09-13T12:06:09.007 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.007 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:06:09.007 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "osd.0", 2024-09-13T12:06:09.007 INFO:tasks.workunit.client.0.smithi032.stdout: "class": "hdd" 2024-09-13T12:06:09.007 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.007 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.007 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 1, 2024-09-13T12:06:09.007 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "osd.1", 2024-09-13T12:06:09.007 INFO:tasks.workunit.client.0.smithi032.stdout: "class": "hdd" 2024-09-13T12:06:09.008 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.008 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.008 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:06:09.008 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "osd.2", 2024-09-13T12:06:09.008 INFO:tasks.workunit.client.0.smithi032.stdout: "class": "hdd" 2024-09-13T12:06:09.008 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:09.008 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:09.008 INFO:tasks.workunit.client.0.smithi032.stdout: "types": [ 2024-09-13T12:06:09.008 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.008 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 0, 2024-09-13T12:06:09.008 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "osd" 2024-09-13T12:06:09.008 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.009 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.009 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 1, 2024-09-13T12:06:09.009 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "host" 2024-09-13T12:06:09.009 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.009 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.009 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 2, 2024-09-13T12:06:09.009 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "chassis" 2024-09-13T12:06:09.009 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.009 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.009 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 3, 2024-09-13T12:06:09.009 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "rack" 2024-09-13T12:06:09.009 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.009 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.009 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 4, 2024-09-13T12:06:09.010 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "row" 2024-09-13T12:06:09.010 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.010 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.010 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 5, 2024-09-13T12:06:09.010 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "pdu" 2024-09-13T12:06:09.010 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.010 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.010 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 6, 2024-09-13T12:06:09.010 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "pod" 2024-09-13T12:06:09.010 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.010 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.010 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 7, 2024-09-13T12:06:09.010 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "room" 2024-09-13T12:06:09.011 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.011 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.011 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 8, 2024-09-13T12:06:09.011 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "datacenter" 2024-09-13T12:06:09.011 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.011 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.011 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 9, 2024-09-13T12:06:09.011 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "zone" 2024-09-13T12:06:09.011 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.011 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.011 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 10, 2024-09-13T12:06:09.011 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "region" 2024-09-13T12:06:09.011 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.012 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.012 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 11, 2024-09-13T12:06:09.012 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "root" 2024-09-13T12:06:09.012 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:09.012 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:09.012 INFO:tasks.workunit.client.0.smithi032.stdout: "buckets": [ 2024-09-13T12:06:09.012 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.012 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -1, 2024-09-13T12:06:09.012 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "default", 2024-09-13T12:06:09.012 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 11, 2024-09-13T12:06:09.012 INFO:tasks.workunit.client.0.smithi032.stdout: "type_name": "root", 2024-09-13T12:06:09.012 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:06:09.012 INFO:tasks.workunit.client.0.smithi032.stdout: "alg": "straw2", 2024-09-13T12:06:09.013 INFO:tasks.workunit.client.0.smithi032.stdout: "hash": "rjenkins1", 2024-09-13T12:06:09.013 INFO:tasks.workunit.client.0.smithi032.stdout: "items": [ 2024-09-13T12:06:09.013 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.013 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -3, 2024-09-13T12:06:09.013 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:06:09.013 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 0 2024-09-13T12:06:09.013 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:09.013 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:09.013 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.013 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.013 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -2, 2024-09-13T12:06:09.013 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "default~hdd", 2024-09-13T12:06:09.013 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 11, 2024-09-13T12:06:09.013 INFO:tasks.workunit.client.0.smithi032.stdout: "type_name": "root", 2024-09-13T12:06:09.014 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:06:09.014 INFO:tasks.workunit.client.0.smithi032.stdout: "alg": "straw2", 2024-09-13T12:06:09.014 INFO:tasks.workunit.client.0.smithi032.stdout: "hash": "rjenkins1", 2024-09-13T12:06:09.014 INFO:tasks.workunit.client.0.smithi032.stdout: "items": [ 2024-09-13T12:06:09.014 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.014 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -4, 2024-09-13T12:06:09.014 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:06:09.014 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 0 2024-09-13T12:06:09.014 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:09.014 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:09.014 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.014 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.014 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -3, 2024-09-13T12:06:09.015 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "smithi032", 2024-09-13T12:06:09.015 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 1, 2024-09-13T12:06:09.015 INFO:tasks.workunit.client.0.smithi032.stdout: "type_name": "host", 2024-09-13T12:06:09.015 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:06:09.015 INFO:tasks.workunit.client.0.smithi032.stdout: "alg": "straw2", 2024-09-13T12:06:09.015 INFO:tasks.workunit.client.0.smithi032.stdout: "hash": "rjenkins1", 2024-09-13T12:06:09.015 INFO:tasks.workunit.client.0.smithi032.stdout: "items": [ 2024-09-13T12:06:09.015 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.015 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:06:09.015 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:06:09.015 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 0 2024-09-13T12:06:09.015 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.015 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.015 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 1, 2024-09-13T12:06:09.016 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:06:09.016 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 1 2024-09-13T12:06:09.016 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.016 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.016 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:06:09.016 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:06:09.016 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 2 2024-09-13T12:06:09.016 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:09.016 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:09.016 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.016 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.016 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -4, 2024-09-13T12:06:09.016 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "smithi032~hdd", 2024-09-13T12:06:09.016 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 1, 2024-09-13T12:06:09.017 INFO:tasks.workunit.client.0.smithi032.stdout: "type_name": "host", 2024-09-13T12:06:09.017 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:06:09.017 INFO:tasks.workunit.client.0.smithi032.stdout: "alg": "straw2", 2024-09-13T12:06:09.017 INFO:tasks.workunit.client.0.smithi032.stdout: "hash": "rjenkins1", 2024-09-13T12:06:09.017 INFO:tasks.workunit.client.0.smithi032.stdout: "items": [ 2024-09-13T12:06:09.017 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.017 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:06:09.017 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:06:09.017 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 0 2024-09-13T12:06:09.017 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.017 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.017 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 1, 2024-09-13T12:06:09.017 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:06:09.017 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 1 2024-09-13T12:06:09.018 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.018 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.018 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:06:09.018 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:06:09.018 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 2 2024-09-13T12:06:09.018 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:09.018 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:09.018 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:09.018 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:09.018 INFO:tasks.workunit.client.0.smithi032.stdout: "rules": [ 2024-09-13T12:06:09.018 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.018 INFO:tasks.workunit.client.0.smithi032.stdout: "rule_id": 0, 2024-09-13T12:06:09.018 INFO:tasks.workunit.client.0.smithi032.stdout: "rule_name": "replicated_rule", 2024-09-13T12:06:09.019 INFO:tasks.workunit.client.0.smithi032.stdout: "type": 1, 2024-09-13T12:06:09.019 INFO:tasks.workunit.client.0.smithi032.stdout: "steps": [ 2024-09-13T12:06:09.019 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.019 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "take", 2024-09-13T12:06:09.019 INFO:tasks.workunit.client.0.smithi032.stdout: "item": -1, 2024-09-13T12:06:09.019 INFO:tasks.workunit.client.0.smithi032.stdout: "item_name": "default" 2024-09-13T12:06:09.019 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.019 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.019 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "choose_firstn", 2024-09-13T12:06:09.019 INFO:tasks.workunit.client.0.smithi032.stdout: "num": 0, 2024-09-13T12:06:09.019 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "osd" 2024-09-13T12:06:09.019 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.019 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.019 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "emit" 2024-09-13T12:06:09.020 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:09.020 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:09.020 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:09.020 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:09.020 INFO:tasks.workunit.client.0.smithi032.stdout: "tunables": { 2024-09-13T12:06:09.020 INFO:tasks.workunit.client.0.smithi032.stdout: "choose_local_tries": 0, 2024-09-13T12:06:09.020 INFO:tasks.workunit.client.0.smithi032.stdout: "choose_local_fallback_tries": 0, 2024-09-13T12:06:09.020 INFO:tasks.workunit.client.0.smithi032.stdout: "choose_total_tries": 50, 2024-09-13T12:06:09.020 INFO:tasks.workunit.client.0.smithi032.stdout: "chooseleaf_descend_once": 1, 2024-09-13T12:06:09.020 INFO:tasks.workunit.client.0.smithi032.stdout: "chooseleaf_vary_r": 1, 2024-09-13T12:06:09.020 INFO:tasks.workunit.client.0.smithi032.stdout: "chooseleaf_stable": 1, 2024-09-13T12:06:09.020 INFO:tasks.workunit.client.0.smithi032.stdout: "msr_descents": 100, 2024-09-13T12:06:09.020 INFO:tasks.workunit.client.0.smithi032.stdout: "msr_collision_tries": 100, 2024-09-13T12:06:09.020 INFO:tasks.workunit.client.0.smithi032.stdout: "straw_calc_version": 1, 2024-09-13T12:06:09.021 INFO:tasks.workunit.client.0.smithi032.stdout: "allowed_bucket_algs": 54, 2024-09-13T12:06:09.021 INFO:tasks.workunit.client.0.smithi032.stdout: "profile": "jewel", 2024-09-13T12:06:09.021 INFO:tasks.workunit.client.0.smithi032.stdout: "optimal_tunables": 1, 2024-09-13T12:06:09.021 INFO:tasks.workunit.client.0.smithi032.stdout: "legacy_tunables": 0, 2024-09-13T12:06:09.021 INFO:tasks.workunit.client.0.smithi032.stdout: "minimum_required_version": "jewel", 2024-09-13T12:06:09.021 INFO:tasks.workunit.client.0.smithi032.stdout: "require_feature_tunables": 1, 2024-09-13T12:06:09.021 INFO:tasks.workunit.client.0.smithi032.stdout: "require_feature_tunables2": 1, 2024-09-13T12:06:09.021 INFO:tasks.workunit.client.0.smithi032.stdout: "has_v2_rules": 0, 2024-09-13T12:06:09.021 INFO:tasks.workunit.client.0.smithi032.stdout: "require_feature_tunables3": 1, 2024-09-13T12:06:09.021 INFO:tasks.workunit.client.0.smithi032.stdout: "has_v3_rules": 0, 2024-09-13T12:06:09.021 INFO:tasks.workunit.client.0.smithi032.stdout: "has_v4_buckets": 1, 2024-09-13T12:06:09.021 INFO:tasks.workunit.client.0.smithi032.stdout: "require_feature_tunables5": 1, 2024-09-13T12:06:09.022 INFO:tasks.workunit.client.0.smithi032.stdout: "has_v5_rules": 0, 2024-09-13T12:06:09.022 INFO:tasks.workunit.client.0.smithi032.stdout: "has_msr_rules": 0 2024-09-13T12:06:09.022 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.022 INFO:tasks.workunit.client.0.smithi032.stdout: "choose_args": {} 2024-09-13T12:06:09.022 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.022 INFO:tasks.workunit.client.0.smithi032.stdout: "fsmap": { 2024-09-13T12:06:09.022 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 1, 2024-09-13T12:06:09.022 INFO:tasks.workunit.client.0.smithi032.stdout: "btime": "2024-09-13T12:02:42:144183+0000", 2024-09-13T12:06:09.022 INFO:tasks.workunit.client.0.smithi032.stdout: "default_fscid": -1, 2024-09-13T12:06:09.022 INFO:tasks.workunit.client.0.smithi032.stdout: "compat": { 2024-09-13T12:06:09.023 INFO:tasks.workunit.client.0.smithi032.stderr:report 37491567 2024-09-13T12:06:09.023 INFO:tasks.workunit.client.0.smithi032.stdout: "compat": {}, 2024-09-13T12:06:09.023 INFO:tasks.workunit.client.0.smithi032.stdout: "ro_compat": {}, 2024-09-13T12:06:09.023 INFO:tasks.workunit.client.0.smithi032.stdout: "incompat": { 2024-09-13T12:06:09.023 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_1": "base v0.20", 2024-09-13T12:06:09.023 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_2": "client writeable ranges", 2024-09-13T12:06:09.023 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_3": "default file layouts on dirs", 2024-09-13T12:06:09.023 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_4": "dir inode in separate object", 2024-09-13T12:06:09.023 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_5": "mds uses versioned encoding", 2024-09-13T12:06:09.024 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-13T12:06:09.024 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_8": "no anchor table", 2024-09-13T12:06:09.024 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_9": "file layout v2", 2024-09-13T12:06:09.024 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_10": "snaprealm v2", 2024-09-13T12:06:09.024 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_11": "minor log segments", 2024-09-13T12:06:09.024 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_12": "quiesce subvolumes" 2024-09-13T12:06:09.024 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:09.024 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.024 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_flags": { 2024-09-13T12:06:09.024 INFO:tasks.workunit.client.0.smithi032.stdout: "enable_multiple": true, 2024-09-13T12:06:09.024 INFO:tasks.workunit.client.0.smithi032.stdout: "ever_enabled_multiple": true 2024-09-13T12:06:09.024 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.024 INFO:tasks.workunit.client.0.smithi032.stdout: "standbys": [], 2024-09-13T12:06:09.025 INFO:tasks.workunit.client.0.smithi032.stdout: "filesystems": [] 2024-09-13T12:06:09.025 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.025 INFO:tasks.workunit.client.0.smithi032.stdout: "mdsmap_first_committed": 1, 2024-09-13T12:06:09.025 INFO:tasks.workunit.client.0.smithi032.stdout: "mdsmap_last_committed": 1, 2024-09-13T12:06:09.025 INFO:tasks.workunit.client.0.smithi032.stdout: "auth": { 2024-09-13T12:06:09.025 INFO:tasks.workunit.client.0.smithi032.stdout: "first_committed": 1, 2024-09-13T12:06:09.025 INFO:tasks.workunit.client.0.smithi032.stdout: "last_committed": 5, 2024-09-13T12:06:09.025 INFO:tasks.workunit.client.0.smithi032.stdout: "num_secrets": 3 2024-09-13T12:06:09.025 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.025 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg": 1, 2024-09-13T12:06:09.025 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg_active": 1, 2024-09-13T12:06:09.025 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg_unknown": 0, 2024-09-13T12:06:09.025 INFO:tasks.workunit.client.0.smithi032.stdout: "num_osd": 3, 2024-09-13T12:06:09.025 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_sum": { 2024-09-13T12:06:09.026 INFO:tasks.workunit.client.0.smithi032.stdout: "stat_sum": { 2024-09-13T12:06:09.026 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes": 48300, 2024-09-13T12:06:09.026 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects": 100, 2024-09-13T12:06:09.026 INFO:tasks.workunit.client.0.smithi032.stdout: "num_object_clones": 0, 2024-09-13T12:06:09.026 INFO:tasks.workunit.client.0.smithi032.stdout: "num_object_copies": 300, 2024-09-13T12:06:09.026 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_missing_on_primary": 0, 2024-09-13T12:06:09.026 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_missing": 0, 2024-09-13T12:06:09.026 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_degraded": 100, 2024-09-13T12:06:09.026 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_misplaced": 0, 2024-09-13T12:06:09.026 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_unfound": 0, 2024-09-13T12:06:09.026 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_dirty": 100, 2024-09-13T12:06:09.026 INFO:tasks.workunit.client.0.smithi032.stdout: "num_whiteouts": 0, 2024-09-13T12:06:09.026 INFO:tasks.workunit.client.0.smithi032.stdout: "num_read": 0, 2024-09-13T12:06:09.026 INFO:tasks.workunit.client.0.smithi032.stdout: "num_read_kb": 0, 2024-09-13T12:06:09.027 INFO:tasks.workunit.client.0.smithi032.stdout: "num_write": 100, 2024-09-13T12:06:09.027 INFO:tasks.workunit.client.0.smithi032.stdout: "num_write_kb": 100, 2024-09-13T12:06:09.027 INFO:tasks.workunit.client.0.smithi032.stdout: "num_scrub_errors": 0, 2024-09-13T12:06:09.027 INFO:tasks.workunit.client.0.smithi032.stdout: "num_shallow_scrub_errors": 0, 2024-09-13T12:06:09.027 INFO:tasks.workunit.client.0.smithi032.stdout: "num_deep_scrub_errors": 0, 2024-09-13T12:06:09.027 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_recovered": 0, 2024-09-13T12:06:09.027 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes_recovered": 0, 2024-09-13T12:06:09.027 INFO:tasks.workunit.client.0.smithi032.stdout: "num_keys_recovered": 0, 2024-09-13T12:06:09.027 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_omap": 0, 2024-09-13T12:06:09.027 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_hit_set_archive": 0, 2024-09-13T12:06:09.027 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes_hit_set_archive": 0, 2024-09-13T12:06:09.027 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush": 0, 2024-09-13T12:06:09.027 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_kb": 0, 2024-09-13T12:06:09.028 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict": 0, 2024-09-13T12:06:09.028 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_kb": 0, 2024-09-13T12:06:09.028 INFO:tasks.workunit.client.0.smithi032.stdout: "num_promote": 0, 2024-09-13T12:06:09.028 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_mode_high": 0, 2024-09-13T12:06:09.028 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_mode_low": 0, 2024-09-13T12:06:09.028 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_mode_some": 0, 2024-09-13T12:06:09.028 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_mode_full": 0, 2024-09-13T12:06:09.028 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_pinned": 0, 2024-09-13T12:06:09.028 INFO:tasks.workunit.client.0.smithi032.stdout: "num_legacy_snapsets": 0, 2024-09-13T12:06:09.028 INFO:tasks.workunit.client.0.smithi032.stdout: "num_large_omap_objects": 0, 2024-09-13T12:06:09.028 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_manifest": 0, 2024-09-13T12:06:09.028 INFO:tasks.workunit.client.0.smithi032.stdout: "num_omap_bytes": 0, 2024-09-13T12:06:09.028 INFO:tasks.workunit.client.0.smithi032.stdout: "num_omap_keys": 0, 2024-09-13T12:06:09.029 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_repaired": 0 2024-09-13T12:06:09.029 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.029 INFO:tasks.workunit.client.0.smithi032.stdout: "store_stats": { 2024-09-13T12:06:09.029 INFO:tasks.workunit.client.0.smithi032.stdout: "total": 0, 2024-09-13T12:06:09.029 INFO:tasks.workunit.client.0.smithi032.stdout: "available": 0, 2024-09-13T12:06:09.029 INFO:tasks.workunit.client.0.smithi032.stdout: "internally_reserved": 0, 2024-09-13T12:06:09.029 INFO:tasks.workunit.client.0.smithi032.stdout: "allocated": 0, 2024-09-13T12:06:09.029 INFO:tasks.workunit.client.0.smithi032.stdout: "data_stored": 0, 2024-09-13T12:06:09.029 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed": 0, 2024-09-13T12:06:09.029 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_allocated": 0, 2024-09-13T12:06:09.029 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_original": 0, 2024-09-13T12:06:09.029 INFO:tasks.workunit.client.0.smithi032.stdout: "omap_allocated": 0, 2024-09-13T12:06:09.029 INFO:tasks.workunit.client.0.smithi032.stdout: "internal_metadata": 0 2024-09-13T12:06:09.029 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.030 INFO:tasks.workunit.client.0.smithi032.stdout: "log_size": 16, 2024-09-13T12:06:09.030 INFO:tasks.workunit.client.0.smithi032.stdout: "ondisk_log_size": 16, 2024-09-13T12:06:09.030 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 2, 2024-09-13T12:06:09.030 INFO:tasks.workunit.client.0.smithi032.stdout: "acting": 2, 2024-09-13T12:06:09.030 INFO:tasks.workunit.client.0.smithi032.stdout: "num_store_stats": 0 2024-09-13T12:06:09.030 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.030 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_sum": { 2024-09-13T12:06:09.030 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 0, 2024-09-13T12:06:09.030 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 0, 2024-09-13T12:06:09.030 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pgs": 3, 2024-09-13T12:06:09.030 INFO:tasks.workunit.client.0.smithi032.stdout: "num_osds": 3, 2024-09-13T12:06:09.030 INFO:tasks.workunit.client.0.smithi032.stdout: "num_per_pool_osds": 3, 2024-09-13T12:06:09.030 INFO:tasks.workunit.client.0.smithi032.stdout: "num_per_pool_omap_osds": 3, 2024-09-13T12:06:09.030 INFO:tasks.workunit.client.0.smithi032.stdout: "kb": 314572800, 2024-09-13T12:06:09.031 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used": 83644, 2024-09-13T12:06:09.031 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_data": 2104, 2024-09-13T12:06:09.031 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_omap": 10, 2024-09-13T12:06:09.031 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_meta": 81461, 2024-09-13T12:06:09.031 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_avail": 314489156, 2024-09-13T12:06:09.031 INFO:tasks.workunit.client.0.smithi032.stdout: "statfs": { 2024-09-13T12:06:09.031 INFO:tasks.workunit.client.0.smithi032.stdout: "total": 322122547200, 2024-09-13T12:06:09.031 INFO:tasks.workunit.client.0.smithi032.stdout: "available": 322036895744, 2024-09-13T12:06:09.031 INFO:tasks.workunit.client.0.smithi032.stdout: "internally_reserved": 0, 2024-09-13T12:06:09.031 INFO:tasks.workunit.client.0.smithi032.stdout: "allocated": 2154496, 2024-09-13T12:06:09.031 INFO:tasks.workunit.client.0.smithi032.stdout: "data_stored": 418637, 2024-09-13T12:06:09.031 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed": 0, 2024-09-13T12:06:09.031 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_allocated": 0, 2024-09-13T12:06:09.032 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_original": 0, 2024-09-13T12:06:09.032 INFO:tasks.workunit.client.0.smithi032.stdout: "omap_allocated": 10397, 2024-09-13T12:06:09.032 INFO:tasks.workunit.client.0.smithi032.stdout: "internal_metadata": 83416931 2024-09-13T12:06:09.032 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.032 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_peers": [], 2024-09-13T12:06:09.032 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_trim_queue_len": 0, 2024-09-13T12:06:09.032 INFO:tasks.workunit.client.0.smithi032.stdout: "num_snap_trimming": 0, 2024-09-13T12:06:09.032 INFO:tasks.workunit.client.0.smithi032.stdout: "num_shards_repaired": 0, 2024-09-13T12:06:09.032 INFO:tasks.workunit.client.0.smithi032.stdout: "op_queue_age_hist": { 2024-09-13T12:06:09.032 INFO:tasks.workunit.client.0.smithi032.stdout: "histogram": [], 2024-09-13T12:06:09.032 INFO:tasks.workunit.client.0.smithi032.stdout: "upper_bound": 1 2024-09-13T12:06:09.032 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.032 INFO:tasks.workunit.client.0.smithi032.stdout: "perf_stat": { 2024-09-13T12:06:09.032 INFO:tasks.workunit.client.0.smithi032.stdout: "commit_latency_ms": 0, 2024-09-13T12:06:09.033 INFO:tasks.workunit.client.0.smithi032.stdout: "apply_latency_ms": 0, 2024-09-13T12:06:09.033 INFO:tasks.workunit.client.0.smithi032.stdout: "commit_latency_ns": 0, 2024-09-13T12:06:09.033 INFO:tasks.workunit.client.0.smithi032.stdout: "apply_latency_ns": 0 2024-09-13T12:06:09.033 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.033 INFO:tasks.workunit.client.0.smithi032.stdout: "alerts": [], 2024-09-13T12:06:09.033 INFO:tasks.workunit.client.0.smithi032.stdout: "network_ping_times": [] 2024-09-13T12:06:09.033 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.033 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_sum_by_class": { 2024-09-13T12:06:09.033 INFO:tasks.workunit.client.0.smithi032.stdout: "hdd": { 2024-09-13T12:06:09.033 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 0, 2024-09-13T12:06:09.033 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 0, 2024-09-13T12:06:09.033 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pgs": 3, 2024-09-13T12:06:09.033 INFO:tasks.workunit.client.0.smithi032.stdout: "num_osds": 3, 2024-09-13T12:06:09.033 INFO:tasks.workunit.client.0.smithi032.stdout: "num_per_pool_osds": 3, 2024-09-13T12:06:09.034 INFO:tasks.workunit.client.0.smithi032.stdout: "num_per_pool_omap_osds": 3, 2024-09-13T12:06:09.034 INFO:tasks.workunit.client.0.smithi032.stdout: "kb": 314572800, 2024-09-13T12:06:09.034 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used": 83644, 2024-09-13T12:06:09.034 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_data": 2104, 2024-09-13T12:06:09.034 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_omap": 10, 2024-09-13T12:06:09.034 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_meta": 81461, 2024-09-13T12:06:09.034 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_avail": 314489156, 2024-09-13T12:06:09.034 INFO:tasks.workunit.client.0.smithi032.stdout: "statfs": { 2024-09-13T12:06:09.034 INFO:tasks.workunit.client.0.smithi032.stdout: "total": 322122547200, 2024-09-13T12:06:09.034 INFO:tasks.workunit.client.0.smithi032.stdout: "available": 322036895744, 2024-09-13T12:06:09.034 INFO:tasks.workunit.client.0.smithi032.stdout: "internally_reserved": 0, 2024-09-13T12:06:09.034 INFO:tasks.workunit.client.0.smithi032.stdout: "allocated": 2154496, 2024-09-13T12:06:09.034 INFO:tasks.workunit.client.0.smithi032.stdout: "data_stored": 418637, 2024-09-13T12:06:09.034 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed": 0, 2024-09-13T12:06:09.035 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_allocated": 0, 2024-09-13T12:06:09.035 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_original": 0, 2024-09-13T12:06:09.035 INFO:tasks.workunit.client.0.smithi032.stdout: "omap_allocated": 10397, 2024-09-13T12:06:09.035 INFO:tasks.workunit.client.0.smithi032.stdout: "internal_metadata": 83416931 2024-09-13T12:06:09.035 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.035 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_peers": [], 2024-09-13T12:06:09.035 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_trim_queue_len": 0, 2024-09-13T12:06:09.035 INFO:tasks.workunit.client.0.smithi032.stdout: "num_snap_trimming": 0, 2024-09-13T12:06:09.035 INFO:tasks.workunit.client.0.smithi032.stdout: "num_shards_repaired": 0, 2024-09-13T12:06:09.035 INFO:tasks.workunit.client.0.smithi032.stdout: "op_queue_age_hist": { 2024-09-13T12:06:09.035 INFO:tasks.workunit.client.0.smithi032.stdout: "histogram": [], 2024-09-13T12:06:09.035 INFO:tasks.workunit.client.0.smithi032.stdout: "upper_bound": 1 2024-09-13T12:06:09.035 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.036 INFO:tasks.workunit.client.0.smithi032.stdout: "perf_stat": { 2024-09-13T12:06:09.036 INFO:tasks.workunit.client.0.smithi032.stdout: "commit_latency_ms": 0, 2024-09-13T12:06:09.036 INFO:tasks.workunit.client.0.smithi032.stdout: "apply_latency_ms": 0, 2024-09-13T12:06:09.036 INFO:tasks.workunit.client.0.smithi032.stdout: "commit_latency_ns": 0, 2024-09-13T12:06:09.036 INFO:tasks.workunit.client.0.smithi032.stdout: "apply_latency_ns": 0 2024-09-13T12:06:09.036 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.036 INFO:tasks.workunit.client.0.smithi032.stdout: "alerts": [], 2024-09-13T12:06:09.036 INFO:tasks.workunit.client.0.smithi032.stdout: "network_ping_times": [] 2024-09-13T12:06:09.036 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:09.036 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.036 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_stats": [ 2024-09-13T12:06:09.036 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.036 INFO:tasks.workunit.client.0.smithi032.stdout: "poolid": 1, 2024-09-13T12:06:09.037 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg": 1, 2024-09-13T12:06:09.037 INFO:tasks.workunit.client.0.smithi032.stdout: "stat_sum": { 2024-09-13T12:06:09.037 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes": 48300, 2024-09-13T12:06:09.037 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects": 100, 2024-09-13T12:06:09.037 INFO:tasks.workunit.client.0.smithi032.stdout: "num_object_clones": 0, 2024-09-13T12:06:09.037 INFO:tasks.workunit.client.0.smithi032.stdout: "num_object_copies": 300, 2024-09-13T12:06:09.037 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_missing_on_primary": 0, 2024-09-13T12:06:09.037 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_missing": 0, 2024-09-13T12:06:09.037 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_degraded": 100, 2024-09-13T12:06:09.037 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_misplaced": 0, 2024-09-13T12:06:09.037 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_unfound": 0, 2024-09-13T12:06:09.037 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_dirty": 100, 2024-09-13T12:06:09.037 INFO:tasks.workunit.client.0.smithi032.stdout: "num_whiteouts": 0, 2024-09-13T12:06:09.037 INFO:tasks.workunit.client.0.smithi032.stdout: "num_read": 0, 2024-09-13T12:06:09.037 INFO:tasks.workunit.client.0.smithi032.stdout: "num_read_kb": 0, 2024-09-13T12:06:09.038 INFO:tasks.workunit.client.0.smithi032.stdout: "num_write": 100, 2024-09-13T12:06:09.038 INFO:tasks.workunit.client.0.smithi032.stdout: "num_write_kb": 100, 2024-09-13T12:06:09.038 INFO:tasks.workunit.client.0.smithi032.stdout: "num_scrub_errors": 0, 2024-09-13T12:06:09.038 INFO:tasks.workunit.client.0.smithi032.stdout: "num_shallow_scrub_errors": 0, 2024-09-13T12:06:09.038 INFO:tasks.workunit.client.0.smithi032.stdout: "num_deep_scrub_errors": 0, 2024-09-13T12:06:09.038 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_recovered": 0, 2024-09-13T12:06:09.038 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes_recovered": 0, 2024-09-13T12:06:09.038 INFO:tasks.workunit.client.0.smithi032.stdout: "num_keys_recovered": 0, 2024-09-13T12:06:09.038 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_omap": 0, 2024-09-13T12:06:09.038 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_hit_set_archive": 0, 2024-09-13T12:06:09.038 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes_hit_set_archive": 0, 2024-09-13T12:06:09.038 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush": 0, 2024-09-13T12:06:09.038 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_kb": 0, 2024-09-13T12:06:09.039 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict": 0, 2024-09-13T12:06:09.039 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_kb": 0, 2024-09-13T12:06:09.039 INFO:tasks.workunit.client.0.smithi032.stdout: "num_promote": 0, 2024-09-13T12:06:09.039 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_mode_high": 0, 2024-09-13T12:06:09.039 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_mode_low": 0, 2024-09-13T12:06:09.039 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_mode_some": 0, 2024-09-13T12:06:09.039 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_mode_full": 0, 2024-09-13T12:06:09.039 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_pinned": 0, 2024-09-13T12:06:09.039 INFO:tasks.workunit.client.0.smithi032.stdout: "num_legacy_snapsets": 0, 2024-09-13T12:06:09.039 INFO:tasks.workunit.client.0.smithi032.stdout: "num_large_omap_objects": 0, 2024-09-13T12:06:09.039 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_manifest": 0, 2024-09-13T12:06:09.039 INFO:tasks.workunit.client.0.smithi032.stdout: "num_omap_bytes": 0, 2024-09-13T12:06:09.039 INFO:tasks.workunit.client.0.smithi032.stdout: "num_omap_keys": 0, 2024-09-13T12:06:09.040 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_repaired": 0 2024-09-13T12:06:09.040 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.040 INFO:tasks.workunit.client.0.smithi032.stdout: "store_stats": { 2024-09-13T12:06:09.040 INFO:tasks.workunit.client.0.smithi032.stdout: "total": 0, 2024-09-13T12:06:09.040 INFO:tasks.workunit.client.0.smithi032.stdout: "available": 0, 2024-09-13T12:06:09.040 INFO:tasks.workunit.client.0.smithi032.stdout: "internally_reserved": 0, 2024-09-13T12:06:09.040 INFO:tasks.workunit.client.0.smithi032.stdout: "allocated": 1220608, 2024-09-13T12:06:09.040 INFO:tasks.workunit.client.0.smithi032.stdout: "data_stored": 143276, 2024-09-13T12:06:09.040 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed": 0, 2024-09-13T12:06:09.040 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_allocated": 0, 2024-09-13T12:06:09.040 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_original": 0, 2024-09-13T12:06:09.040 INFO:tasks.workunit.client.0.smithi032.stdout: "omap_allocated": 0, 2024-09-13T12:06:09.040 INFO:tasks.workunit.client.0.smithi032.stdout: "internal_metadata": 0 2024-09-13T12:06:09.041 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.041 INFO:tasks.workunit.client.0.smithi032.stdout: "log_size": 16, 2024-09-13T12:06:09.041 INFO:tasks.workunit.client.0.smithi032.stdout: "ondisk_log_size": 16, 2024-09-13T12:06:09.041 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 2, 2024-09-13T12:06:09.041 INFO:tasks.workunit.client.0.smithi032.stdout: "acting": 2, 2024-09-13T12:06:09.041 INFO:tasks.workunit.client.0.smithi032.stdout: "num_store_stats": 3 2024-09-13T12:06:09.041 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:09.041 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:09.041 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_stats": [ 2024-09-13T12:06:09.041 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.041 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 0, 2024-09-13T12:06:09.041 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 154618822667 2024-09-13T12:06:09.042 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.042 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.042 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 1, 2024-09-13T12:06:09.042 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 42949672991 2024-09-13T12:06:09.042 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.042 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.042 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2, 2024-09-13T12:06:09.042 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 163208757258 2024-09-13T12:06:09.042 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:09.042 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:09.042 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg_by_state": [ 2024-09-13T12:06:09.042 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.042 INFO:tasks.workunit.client.0.smithi032.stdout: "state": "active+undersized+degraded", 2024-09-13T12:06:09.043 INFO:tasks.workunit.client.0.smithi032.stdout: "num": 1 2024-09-13T12:06:09.043 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:09.043 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:09.043 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg_by_osd": [ 2024-09-13T12:06:09.043 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.043 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2, 2024-09-13T12:06:09.043 INFO:tasks.workunit.client.0.smithi032.stdout: "num_primary_pg": 0, 2024-09-13T12:06:09.043 INFO:tasks.workunit.client.0.smithi032.stdout: "num_acting_pg": 1, 2024-09-13T12:06:09.043 INFO:tasks.workunit.client.0.smithi032.stdout: "num_up_not_acting_pg": 0 2024-09-13T12:06:09.043 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.043 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.043 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 0, 2024-09-13T12:06:09.043 INFO:tasks.workunit.client.0.smithi032.stdout: "num_primary_pg": 1, 2024-09-13T12:06:09.044 INFO:tasks.workunit.client.0.smithi032.stdout: "num_acting_pg": 1, 2024-09-13T12:06:09.044 INFO:tasks.workunit.client.0.smithi032.stdout: "num_up_not_acting_pg": 0 2024-09-13T12:06:09.044 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.044 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.044 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 1, 2024-09-13T12:06:09.044 INFO:tasks.workunit.client.0.smithi032.stdout: "num_primary_pg": 0, 2024-09-13T12:06:09.044 INFO:tasks.workunit.client.0.smithi032.stdout: "num_acting_pg": 0, 2024-09-13T12:06:09.044 INFO:tasks.workunit.client.0.smithi032.stdout: "num_up_not_acting_pg": 0 2024-09-13T12:06:09.044 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:09.044 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:09.044 INFO:tasks.workunit.client.0.smithi032.stdout: "purged_snaps": [ 2024-09-13T12:06:09.044 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:09.044 INFO:tasks.workunit.client.0.smithi032.stdout: "pool": 1, 2024-09-13T12:06:09.045 INFO:tasks.workunit.client.0.smithi032.stdout: "purged_snaps": {} 2024-09-13T12:06:09.045 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:09.045 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:09.045 INFO:tasks.workunit.client.0.smithi032.stdout: "servicemap": { 2024-09-13T12:06:09.045 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 4, 2024-09-13T12:06:09.045 INFO:tasks.workunit.client.0.smithi032.stdout: "modified": "2024-09-13T12:04:53.122549+0000", 2024-09-13T12:06:09.045 INFO:tasks.workunit.client.0.smithi032.stdout: "services": { 2024-09-13T12:06:09.045 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": { 2024-09-13T12:06:09.045 INFO:tasks.workunit.client.0.smithi032.stdout: "daemons": { 2024-09-13T12:06:09.045 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": "", 2024-09-13T12:06:09.045 INFO:tasks.workunit.client.0.smithi032.stdout: "2": { 2024-09-13T12:06:09.045 INFO:tasks.workunit.client.0.smithi032.stdout: "start_epoch": 0, 2024-09-13T12:06:09.045 INFO:tasks.workunit.client.0.smithi032.stdout: "start_stamp": "0.000000", 2024-09-13T12:06:09.045 INFO:tasks.workunit.client.0.smithi032.stdout: "gid": 0, 2024-09-13T12:06:09.046 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "(unrecognized address family 0)/0", 2024-09-13T12:06:09.046 INFO:tasks.workunit.client.0.smithi032.stdout: "metadata": {}, 2024-09-13T12:06:09.046 INFO:tasks.workunit.client.0.smithi032.stdout: "task_status": {} 2024-09-13T12:06:09.046 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:09.046 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:09.046 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:09.046 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:09.046 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:09.046 INFO:tasks.workunit.client.0.smithi032.stdout: "mgrstat_first_committed": 1, 2024-09-13T12:06:09.046 INFO:tasks.workunit.client.0.smithi032.stdout: "mgrstat_last_committed": 92, 2024-09-13T12:06:09.046 INFO:tasks.workunit.client.0.smithi032.stdout: "logm_first_committed": 1, 2024-09-13T12:06:09.046 INFO:tasks.workunit.client.0.smithi032.stdout: "logm_last_committed": 383, 2024-09-13T12:06:09.046 INFO:tasks.workunit.client.0.smithi032.stdout: "paxos": { 2024-09-13T12:06:09.046 INFO:tasks.workunit.client.0.smithi032.stdout: "first_committed": 1, 2024-09-13T12:06:09.047 INFO:tasks.workunit.client.0.smithi032.stdout: "last_committed": 522, 2024-09-13T12:06:09.047 INFO:tasks.workunit.client.0.smithi032.stdout: "last_pn": 0, 2024-09-13T12:06:09.047 INFO:tasks.workunit.client.0.smithi032.stdout: "accepted_pn": 0 2024-09-13T12:06:09.047 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:09.047 INFO:tasks.workunit.client.0.smithi032.stdout:} 2024-09-13T12:06:09.047 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-13T12:06:09.047 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2024-09-13T12:06:09.047 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2024-09-13T12:06:09.048 INFO:tasks.workunit.client.0.smithi032.stdout:writing non-divergent object existing_4 2024-09-13T12:06:09.048 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2024-09-13T12:06:09.048 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2024-09-13T12:06:09.350 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:06:09.350 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 100 0 100 0 0 48300 0 0 16 84 16 active+undersized+degraded 2024-09-13T12:05:31.913985+0000 29'100 41:130 [0,2] 0 [0,2] 0 0'0 2024-09-13T12:04:30.066569+0000 0'0 2024-09-13T12:04:30.066569+0000 0 0 periodic scrub scheduled @ 2024-09-14T13:20:53.243442+0000 0 0 2024-09-13T12:06:09.350 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:06:09.350 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:06:09.350 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:06:09.360 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2024-09-13T12:06:09.458 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-13T12:06:09.458 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2024-09-13T12:06:09.458 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T12:06:09.458 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T12:06:09.458 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T12:06:09.458 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-13T12:06:09.459 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T12:06:09.459 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T12:06:09.459 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T12:06:09.459 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T12:06:09.459 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T12:06:09.515 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-13T12:06:09.515 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T12:06:09.515 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T12:06:09.515 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T12:06:09.515 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T12:06:09.515 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T12:06:09.961 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T12:06:09.961 INFO:tasks.workunit.client.0.smithi032.stderr:1 2024-09-13T12:06:09.961 INFO:tasks.workunit.client.0.smithi032.stderr:2' 2024-09-13T12:06:09.961 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T12:06:09.961 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:06:09.961 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T12:06:10.085 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822670 2024-09-13T12:06:10.086 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822670 2024-09-13T12:06:10.086 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822670' 2024-09-13T12:06:10.086 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:06:10.086 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T12:06:10.185 INFO:tasks.workunit.client.0.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-13T12:06:10.186 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-13T12:06:10.186 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-13T12:06:10.187 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-13T12:06:10.187 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:06:10.187 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T12:06:10.312 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757261 2024-09-13T12:06:10.312 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757261 2024-09-13T12:06:10.312 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822670 2-163208757261' 2024-09-13T12:06:10.312 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:06:10.313 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-154618822670 2024-09-13T12:06:10.313 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:06:10.313 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T12:06:10.314 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-154618822670 2024-09-13T12:06:10.314 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:06:10.315 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822670 2024-09-13T12:06:10.315 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.0 seq 154618822670 2024-09-13T12:06:10.316 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 154618822670' 2024-09-13T12:06:10.316 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:06:10.754 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822667 -lt 154618822670 2024-09-13T12:06:10.754 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T12:06:11.755 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T12:06:11.755 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:06:12.189 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822670 -lt 154618822670 2024-09-13T12:06:12.189 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:06:12.189 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-163208757261 2024-09-13T12:06:12.189 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:06:12.190 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T12:06:12.191 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-163208757261 2024-09-13T12:06:12.191 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:06:12.192 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757261 2024-09-13T12:06:12.192 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 163208757261' 2024-09-13T12:06:12.192 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.2 seq 163208757261 2024-09-13T12:06:12.192 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T12:06:12.673 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757261 -lt 163208757261 2024-09-13T12:06:12.674 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T12:06:12.674 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:06:12.674 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:06:13.227 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T12:06:13.227 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:06:13.227 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:06:13.227 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:06:13.227 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:06:13.227 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:06:13.228 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:06:13.228 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:06:13.633 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:06:13.634 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:06:13.634 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:06:13.634 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:06:14.195 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:06:14.195 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-13T12:06:14.195 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T12:06:14.195 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-13T12:06:14.195 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:06:14.195 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T12:06:14.296 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:06:14.296 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:06:14.296 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:06:14.296 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:06:14.296 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:06:14.297 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:06:14.297 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:06:14.297 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:06:14.698 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:06:14.698 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:06:14.698 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:06:14.698 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:06:15.261 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:06:15.261 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:06:15.262 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:06:15.262 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:06:15.262 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:06:15.262 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:06:15.262 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:06:15.262 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:06:15.262 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:06:15.861 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:06:15.861 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:06:15.861 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-13T12:06:15.861 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:06:15.861 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T12:06:16.062 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:06:16.062 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:06:16.062 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:06:16.062 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:06:16.063 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:06:16.063 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:06:16.063 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:06:16.063 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:06:16.459 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:06:16.459 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:06:16.460 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:06:16.460 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:06:17.018 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:06:17.019 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:06:17.019 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:06:17.019 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:06:17.019 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:06:17.019 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:06:17.019 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:06:17.019 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:06:17.019 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:06:17.593 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:06:17.593 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:06:17.593 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-13T12:06:17.593 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:06:17.594 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-13T12:06:17.995 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:06:17.995 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:06:17.995 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:06:17.995 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:06:17.995 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:06:17.995 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:06:17.995 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:06:17.995 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:06:18.412 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:06:18.413 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:06:18.413 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:06:18.413 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:06:18.975 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:06:18.975 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:06:18.975 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:06:18.975 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:06:18.975 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:06:18.975 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:06:18.975 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:06:18.976 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:06:18.976 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:06:19.537 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:06:19.538 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:06:19.538 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-13T12:06:19.538 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:06:19.538 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-13T12:06:20.341 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:06:20.342 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:06:20.342 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:06:20.342 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:06:20.342 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:06:20.342 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:06:20.342 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:06:20.343 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:06:20.749 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:06:20.750 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:06:20.750 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:06:20.750 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:06:21.313 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:06:21.313 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:06:21.314 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:06:21.314 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:06:21.314 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:06:21.314 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:06:21.314 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:06:21.314 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:06:21.314 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:06:21.882 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:06:21.882 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:06:21.882 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-13T12:06:21.882 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:06:21.882 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-13T12:06:23.483 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:06:23.483 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:06:23.484 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:06:23.484 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:06:23.484 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:06:23.484 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:06:23.484 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:06:23.485 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:06:23.901 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:06:23.901 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:06:23.902 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:06:23.902 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:06:24.460 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:06:24.460 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:06:24.460 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:06:24.460 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:06:24.460 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:06:24.460 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:06:24.460 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:06:24.460 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:06:24.460 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:06:25.033 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:06:25.033 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:06:25.033 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-13T12:06:25.033 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:06:25.033 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-13T12:06:28.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:06:28.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:06:28.235 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:06:28.235 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:06:28.235 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:06:28.235 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:06:28.235 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:06:28.235 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:06:28.647 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:06:28.647 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:06:28.647 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:06:28.647 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:06:29.197 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:06:29.197 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:06:29.197 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:06:29.197 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:06:29.197 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:06:29.197 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:06:29.198 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:06:29.198 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:06:29.198 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:06:29.751 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:06:29.751 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:06:29.751 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-13T12:06:29.751 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:06:29.751 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-13T12:06:36.152 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:06:36.153 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:06:36.153 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:06:36.153 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:06:36.153 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:06:36.153 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:06:36.153 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:06:36.154 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:06:36.562 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:06:36.563 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:06:36.563 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:06:36.563 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:06:37.118 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:06:37.118 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:06:37.118 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:06:37.118 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:06:37.118 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:06:37.118 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:06:37.118 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:06:37.119 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:06:37.119 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:06:37.702 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:06:37.702 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:06:37.702 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-13T12:06:37.702 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:06:37.702 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-13T12:06:45.003 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:06:45.004 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:06:45.004 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:06:45.004 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:06:45.004 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:06:45.004 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:06:45.004 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:06:45.005 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:06:45.425 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:06:45.425 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:06:45.425 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:06:45.425 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:06:45.979 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:06:45.979 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:06:45.979 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:06:45.979 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:06:45.979 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:06:45.979 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:06:45.979 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:06:45.979 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:06:45.980 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:06:46.537 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:06:46.537 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:06:46.538 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-13T12:06:46.538 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-13T12:06:47.114 INFO:tasks.workunit.client.0.smithi032.stdout:{ 2024-09-13T12:06:47.114 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_fingerprint": "113f00b8-a374-45c4-82e1-e7adfdcf405d", 2024-09-13T12:06:47.114 INFO:tasks.workunit.client.0.smithi032.stdout: "version": "19.3.0-4697-gd58f7306", 2024-09-13T12:06:47.114 INFO:tasks.workunit.client.0.smithi032.stdout: "commit": "d58f73069a8157a9ace12a0247ee3e8497707ea9", 2024-09-13T12:06:47.114 INFO:tasks.workunit.client.0.smithi032.stdout: "timestamp": "2024-09-13T12:06:47.112587+0000", 2024-09-13T12:06:47.114 INFO:tasks.workunit.client.0.smithi032.stdout: "tag": "", 2024-09-13T12:06:47.114 INFO:tasks.workunit.client.0.smithi032.stdout: "health": { 2024-09-13T12:06:47.114 INFO:tasks.workunit.client.0.smithi032.stdout: "status": "HEALTH_WARN", 2024-09-13T12:06:47.114 INFO:tasks.workunit.client.0.smithi032.stdout: "checks": { 2024-09-13T12:06:47.114 INFO:tasks.workunit.client.0.smithi032.stdout: "OSDMAP_FLAGS": { 2024-09-13T12:06:47.115 INFO:tasks.workunit.client.0.smithi032.stdout: "severity": "HEALTH_WARN", 2024-09-13T12:06:47.115 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": { 2024-09-13T12:06:47.115 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-13T12:06:47.115 INFO:tasks.workunit.client.0.smithi032.stdout: "count": 17 2024-09-13T12:06:47.115 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.115 INFO:tasks.workunit.client.0.smithi032.stdout: "detail": [], 2024-09-13T12:06:47.115 INFO:tasks.workunit.client.0.smithi032.stdout: "muted": false 2024-09-13T12:06:47.115 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.115 INFO:tasks.workunit.client.0.smithi032.stdout: "OSD_DOWN": { 2024-09-13T12:06:47.115 INFO:tasks.workunit.client.0.smithi032.stdout: "severity": "HEALTH_WARN", 2024-09-13T12:06:47.115 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": { 2024-09-13T12:06:47.115 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "1 osds down", 2024-09-13T12:06:47.116 INFO:tasks.workunit.client.0.smithi032.stdout: "count": 1 2024-09-13T12:06:47.116 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.116 INFO:tasks.workunit.client.0.smithi032.stdout: "detail": [ 2024-09-13T12:06:47.116 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.116 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "osd.1 (root=default,host=smithi032) is down" 2024-09-13T12:06:47.116 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.116 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:47.116 INFO:tasks.workunit.client.0.smithi032.stdout: "muted": false 2024-09-13T12:06:47.116 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.116 INFO:tasks.workunit.client.0.smithi032.stdout: "PG_DEGRADED": { 2024-09-13T12:06:47.116 INFO:tasks.workunit.client.0.smithi032.stdout: "severity": "HEALTH_WARN", 2024-09-13T12:06:47.116 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": { 2024-09-13T12:06:47.116 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2024-09-13T12:06:47.117 INFO:tasks.workunit.client.0.smithi032.stdout: "count": 2 2024-09-13T12:06:47.117 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.117 INFO:tasks.workunit.client.0.smithi032.stdout: "detail": [ 2024-09-13T12:06:47.117 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.117 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.0 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-13T12:06:47.117 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.117 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:47.117 INFO:tasks.workunit.client.0.smithi032.stdout: "muted": false 2024-09-13T12:06:47.117 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.117 INFO:tasks.workunit.client.0.smithi032.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-13T12:06:47.117 INFO:tasks.workunit.client.0.smithi032.stdout: "severity": "HEALTH_WARN", 2024-09-13T12:06:47.117 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": { 2024-09-13T12:06:47.117 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-13T12:06:47.118 INFO:tasks.workunit.client.0.smithi032.stdout: "count": 1 2024-09-13T12:06:47.118 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.118 INFO:tasks.workunit.client.0.smithi032.stdout: "detail": [ 2024-09-13T12:06:47.118 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.118 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "application not enabled on pool 'test'" 2024-09-13T12:06:47.118 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.118 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.118 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-13T12:06:47.118 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.118 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:47.118 INFO:tasks.workunit.client.0.smithi032.stdout: "muted": false 2024-09-13T12:06:47.118 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.119 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.119 INFO:tasks.workunit.client.0.smithi032.stdout: "mutes": [] 2024-09-13T12:06:47.119 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.119 INFO:tasks.workunit.client.0.smithi032.stdout: "monmap_first_committed": 1, 2024-09-13T12:06:47.119 INFO:tasks.workunit.client.0.smithi032.stdout: "monmap_last_committed": 1, 2024-09-13T12:06:47.120 INFO:tasks.workunit.client.0.smithi032.stdout: "monmap": { 2024-09-13T12:06:47.120 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 1, 2024-09-13T12:06:47.120 INFO:tasks.workunit.client.0.smithi032.stdout: "fsid": "e12dfbc3-9bf5-45db-a6f0-f25db93ee351", 2024-09-13T12:06:47.120 INFO:tasks.workunit.client.0.smithi032.stdout: "modified": "2024-09-13T12:02:41.228021Z", 2024-09-13T12:06:47.120 INFO:tasks.workunit.client.0.smithi032.stdout: "created": "2024-09-13T12:02:41.228021Z", 2024-09-13T12:06:47.120 INFO:tasks.workunit.client.0.smithi032.stdout: "min_mon_release": 19, 2024-09-13T12:06:47.120 INFO:tasks.workunit.client.0.smithi032.stdout: "min_mon_release_name": "squid", 2024-09-13T12:06:47.120 INFO:tasks.workunit.client.0.smithi032.stdout: "election_strategy": 1, 2024-09-13T12:06:47.120 INFO:tasks.workunit.client.0.smithi032.stdout: "disallowed_leaders: ": "", 2024-09-13T12:06:47.120 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_mode": false, 2024-09-13T12:06:47.120 INFO:tasks.workunit.client.0.smithi032.stdout: "tiebreaker_mon": "", 2024-09-13T12:06:47.120 INFO:tasks.workunit.client.0.smithi032.stdout: "removed_ranks: ": "", 2024-09-13T12:06:47.121 INFO:tasks.workunit.client.0.smithi032.stdout: "features": { 2024-09-13T12:06:47.121 INFO:tasks.workunit.client.0.smithi032.stdout: "persistent": [ 2024-09-13T12:06:47.121 INFO:tasks.workunit.client.0.smithi032.stdout: "kraken", 2024-09-13T12:06:47.121 INFO:tasks.workunit.client.0.smithi032.stdout: "luminous", 2024-09-13T12:06:47.121 INFO:tasks.workunit.client.0.smithi032.stdout: "mimic", 2024-09-13T12:06:47.121 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap-prune", 2024-09-13T12:06:47.121 INFO:tasks.workunit.client.0.smithi032.stdout: "nautilus", 2024-09-13T12:06:47.121 INFO:tasks.workunit.client.0.smithi032.stdout: "octopus", 2024-09-13T12:06:47.122 INFO:tasks.workunit.client.0.smithi032.stdout: "pacific", 2024-09-13T12:06:47.122 INFO:tasks.workunit.client.0.smithi032.stdout: "elector-pinging", 2024-09-13T12:06:47.122 INFO:tasks.workunit.client.0.smithi032.stdout: "quincy", 2024-09-13T12:06:47.122 INFO:tasks.workunit.client.0.smithi032.stdout: "reef", 2024-09-13T12:06:47.122 INFO:tasks.workunit.client.0.smithi032.stdout: "squid" 2024-09-13T12:06:47.122 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:47.122 INFO:tasks.workunit.client.0.smithi032.stdout: "optional": [] 2024-09-13T12:06:47.122 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.122 INFO:tasks.workunit.client.0.smithi032.stdout: "mons": [ 2024-09-13T12:06:47.122 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.122 INFO:tasks.workunit.client.0.smithi032.stdout: "rank": 0, 2024-09-13T12:06:47.122 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "a", 2024-09-13T12:06:47.123 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addrs": { 2024-09-13T12:06:47.123 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:06:47.123 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.123 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:06:47.123 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:7115", 2024-09-13T12:06:47.123 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 0 2024-09-13T12:06:47.123 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.123 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:47.123 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.123 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:7115/0", 2024-09-13T12:06:47.124 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-13T12:06:47.124 INFO:tasks.workunit.client.0.smithi032.stdout: "priority": 0, 2024-09-13T12:06:47.124 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 0, 2024-09-13T12:06:47.124 INFO:tasks.workunit.client.0.smithi032.stdout: "crush_location": "{}" 2024-09-13T12:06:47.124 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.124 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:47.124 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.124 INFO:tasks.workunit.client.0.smithi032.stdout: "quorum": [ 2024-09-13T12:06:47.125 INFO:tasks.workunit.client.0.smithi032.stdout: 0 2024-09-13T12:06:47.125 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:47.125 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap": { 2024-09-13T12:06:47.125 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 41, 2024-09-13T12:06:47.125 INFO:tasks.workunit.client.0.smithi032.stdout: "fsid": "e12dfbc3-9bf5-45db-a6f0-f25db93ee351", 2024-09-13T12:06:47.125 INFO:tasks.workunit.client.0.smithi032.stdout: "created": "2024-09-13T12:02:42.168711+0000", 2024-09-13T12:06:47.125 INFO:tasks.workunit.client.0.smithi032.stdout: "modified": "2024-09-13T12:05:31.594789+0000", 2024-09-13T12:06:47.125 INFO:tasks.workunit.client.0.smithi032.stdout: "last_up_change": "2024-09-13T12:05:29.113086+0000", 2024-09-13T12:06:47.125 INFO:tasks.workunit.client.0.smithi032.stdout: "last_in_change": "2024-09-13T12:05:31.074457+0000", 2024-09-13T12:06:47.125 INFO:tasks.workunit.client.0.smithi032.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T12:06:47.125 INFO:tasks.workunit.client.0.smithi032.stdout: "flags_num": 5800384, 2024-09-13T12:06:47.126 INFO:tasks.workunit.client.0.smithi032.stdout: "flags_set": [ 2024-09-13T12:06:47.126 INFO:tasks.workunit.client.0.smithi032.stdout: "nodown", 2024-09-13T12:06:47.126 INFO:tasks.workunit.client.0.smithi032.stdout: "noin", 2024-09-13T12:06:47.126 INFO:tasks.workunit.client.0.smithi032.stdout: "noout", 2024-09-13T12:06:47.126 INFO:tasks.workunit.client.0.smithi032.stdout: "pglog_hardlimit", 2024-09-13T12:06:47.126 INFO:tasks.workunit.client.0.smithi032.stdout: "purged_snapdirs", 2024-09-13T12:06:47.126 INFO:tasks.workunit.client.0.smithi032.stdout: "recovery_deletes", 2024-09-13T12:06:47.126 INFO:tasks.workunit.client.0.smithi032.stdout: "sortbitwise" 2024-09-13T12:06:47.126 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:47.126 INFO:tasks.workunit.client.0.smithi032.stdout: "crush_version": 9, 2024-09-13T12:06:47.126 INFO:tasks.workunit.client.0.smithi032.stdout: "full_ratio": 0.99000000953674316, 2024-09-13T12:06:47.126 INFO:tasks.workunit.client.0.smithi032.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-13T12:06:47.126 INFO:tasks.workunit.client.0.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T12:06:47.126 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_snapshot": "", 2024-09-13T12:06:47.127 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_max": 1, 2024-09-13T12:06:47.127 INFO:tasks.workunit.client.0.smithi032.stdout: "max_osd": 3, 2024-09-13T12:06:47.127 INFO:tasks.workunit.client.0.smithi032.stdout: "require_min_compat_client": "luminous", 2024-09-13T12:06:47.127 INFO:tasks.workunit.client.0.smithi032.stdout: "min_compat_client": "jewel", 2024-09-13T12:06:47.127 INFO:tasks.workunit.client.0.smithi032.stdout: "require_osd_release": "squid", 2024-09-13T12:06:47.127 INFO:tasks.workunit.client.0.smithi032.stdout: "allow_crimson": false, 2024-09-13T12:06:47.127 INFO:tasks.workunit.client.0.smithi032.stdout: "pools": [ 2024-09-13T12:06:47.127 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.127 INFO:tasks.workunit.client.0.smithi032.stdout: "pool": 1, 2024-09-13T12:06:47.127 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_name": "test", 2024-09-13T12:06:47.127 INFO:tasks.workunit.client.0.smithi032.stdout: "create_time": "2024-09-13T12:04:29.912487+0000", 2024-09-13T12:06:47.127 INFO:tasks.workunit.client.0.smithi032.stdout: "flags": 1, 2024-09-13T12:06:47.127 INFO:tasks.workunit.client.0.smithi032.stdout: "flags_names": "hashpspool", 2024-09-13T12:06:47.128 INFO:tasks.workunit.client.0.smithi032.stdout: "type": 1, 2024-09-13T12:06:47.128 INFO:tasks.workunit.client.0.smithi032.stdout: "size": 3, 2024-09-13T12:06:47.128 INFO:tasks.workunit.client.0.smithi032.stdout: "min_size": 2, 2024-09-13T12:06:47.128 INFO:tasks.workunit.client.0.smithi032.stdout: "crush_rule": 0, 2024-09-13T12:06:47.128 INFO:tasks.workunit.client.0.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-13T12:06:47.128 INFO:tasks.workunit.client.0.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-13T12:06:47.128 INFO:tasks.workunit.client.0.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T12:06:47.128 INFO:tasks.workunit.client.0.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T12:06:47.128 INFO:tasks.workunit.client.0.smithi032.stdout: "is_stretch_pool": false, 2024-09-13T12:06:47.128 INFO:tasks.workunit.client.0.smithi032.stdout: "object_hash": 2, 2024-09-13T12:06:47.129 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_autoscale_mode": "off", 2024-09-13T12:06:47.129 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_num": 1, 2024-09-13T12:06:47.129 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_placement_num": 1, 2024-09-13T12:06:47.129 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_placement_num_target": 1, 2024-09-13T12:06:47.129 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_num_target": 1, 2024-09-13T12:06:47.129 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_num_pending": 1, 2024-09-13T12:06:47.129 INFO:tasks.workunit.client.0.smithi032.stdout: "last_pg_merge_meta": { 2024-09-13T12:06:47.129 INFO:tasks.workunit.client.0.smithi032.stdout: "source_pgid": "0.0", 2024-09-13T12:06:47.129 INFO:tasks.workunit.client.0.smithi032.stdout: "ready_epoch": 0, 2024-09-13T12:06:47.129 INFO:tasks.workunit.client.0.smithi032.stdout: "last_epoch_started": 0, 2024-09-13T12:06:47.129 INFO:tasks.workunit.client.0.smithi032.stdout: "last_epoch_clean": 0, 2024-09-13T12:06:47.129 INFO:tasks.workunit.client.0.smithi032.stdout: "source_version": "0'0", 2024-09-13T12:06:47.129 INFO:tasks.workunit.client.0.smithi032.stdout: "target_version": "0'0" 2024-09-13T12:06:47.130 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.130 INFO:tasks.workunit.client.0.smithi032.stdout: "last_change": "29", 2024-09-13T12:06:47.130 INFO:tasks.workunit.client.0.smithi032.stdout: "last_force_op_resend": "0", 2024-09-13T12:06:47.130 INFO:tasks.workunit.client.0.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T12:06:47.130 INFO:tasks.workunit.client.0.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T12:06:47.130 INFO:tasks.workunit.client.0.smithi032.stdout: "auid": 0, 2024-09-13T12:06:47.130 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_mode": "selfmanaged", 2024-09-13T12:06:47.130 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_seq": 0, 2024-09-13T12:06:47.130 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_epoch": 0, 2024-09-13T12:06:47.130 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_snaps": [], 2024-09-13T12:06:47.130 INFO:tasks.workunit.client.0.smithi032.stdout: "removed_snaps": "[]", 2024-09-13T12:06:47.130 INFO:tasks.workunit.client.0.smithi032.stdout: "quota_max_bytes": 0, 2024-09-13T12:06:47.130 INFO:tasks.workunit.client.0.smithi032.stdout: "quota_max_objects": 0, 2024-09-13T12:06:47.131 INFO:tasks.workunit.client.0.smithi032.stdout: "tiers": [], 2024-09-13T12:06:47.131 INFO:tasks.workunit.client.0.smithi032.stdout: "tier_of": -1, 2024-09-13T12:06:47.131 INFO:tasks.workunit.client.0.smithi032.stdout: "read_tier": -1, 2024-09-13T12:06:47.131 INFO:tasks.workunit.client.0.smithi032.stdout: "write_tier": -1, 2024-09-13T12:06:47.131 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_mode": "none", 2024-09-13T12:06:47.131 INFO:tasks.workunit.client.0.smithi032.stdout: "target_max_bytes": 0, 2024-09-13T12:06:47.131 INFO:tasks.workunit.client.0.smithi032.stdout: "target_max_objects": 0, 2024-09-13T12:06:47.131 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T12:06:47.131 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T12:06:47.131 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T12:06:47.131 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-13T12:06:47.131 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-13T12:06:47.131 INFO:tasks.workunit.client.0.smithi032.stdout: "erasure_code_profile": "", 2024-09-13T12:06:47.131 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_params": { 2024-09-13T12:06:47.132 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "none" 2024-09-13T12:06:47.132 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.132 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_period": 0, 2024-09-13T12:06:47.132 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_count": 0, 2024-09-13T12:06:47.132 INFO:tasks.workunit.client.0.smithi032.stdout: "use_gmt_hitset": true, 2024-09-13T12:06:47.132 INFO:tasks.workunit.client.0.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-13T12:06:47.132 INFO:tasks.workunit.client.0.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-13T12:06:47.132 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T12:06:47.132 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-13T12:06:47.132 INFO:tasks.workunit.client.0.smithi032.stdout: "grade_table": [], 2024-09-13T12:06:47.132 INFO:tasks.workunit.client.0.smithi032.stdout: "stripe_width": 0, 2024-09-13T12:06:47.132 INFO:tasks.workunit.client.0.smithi032.stdout: "expected_num_objects": 0, 2024-09-13T12:06:47.132 INFO:tasks.workunit.client.0.smithi032.stdout: "fast_read": false, 2024-09-13T12:06:47.133 INFO:tasks.workunit.client.0.smithi032.stdout: "options": {}, 2024-09-13T12:06:47.133 INFO:tasks.workunit.client.0.smithi032.stdout: "application_metadata": {}, 2024-09-13T12:06:47.133 INFO:tasks.workunit.client.0.smithi032.stdout: "read_balance": { 2024-09-13T12:06:47.133 INFO:tasks.workunit.client.0.smithi032.stdout: "score_type": "Fair distribution", 2024-09-13T12:06:47.133 INFO:tasks.workunit.client.0.smithi032.stdout: "score_acting": 2.9900000095367432, 2024-09-13T12:06:47.133 INFO:tasks.workunit.client.0.smithi032.stdout: "score_stable": 2.9900000095367432, 2024-09-13T12:06:47.133 INFO:tasks.workunit.client.0.smithi032.stdout: "optimal_score": 0.67000001668930054, 2024-09-13T12:06:47.133 INFO:tasks.workunit.client.0.smithi032.stdout: "raw_score_acting": 2, 2024-09-13T12:06:47.133 INFO:tasks.workunit.client.0.smithi032.stdout: "raw_score_stable": 2, 2024-09-13T12:06:47.133 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_affinity_weighted": 1, 2024-09-13T12:06:47.133 INFO:tasks.workunit.client.0.smithi032.stdout: "average_primary_affinity": 1, 2024-09-13T12:06:47.133 INFO:tasks.workunit.client.0.smithi032.stdout: "average_primary_affinity_weighted": 1 2024-09-13T12:06:47.134 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.134 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.134 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:47.134 INFO:tasks.workunit.client.0.smithi032.stdout: "osds": [ 2024-09-13T12:06:47.134 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.134 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 0, 2024-09-13T12:06:47.134 INFO:tasks.workunit.client.0.smithi032.stdout: "uuid": "940cd0ac-d018-458e-a8bb-6eb19147770c", 2024-09-13T12:06:47.134 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 1, 2024-09-13T12:06:47.134 INFO:tasks.workunit.client.0.smithi032.stdout: "in": 1, 2024-09-13T12:06:47.134 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 1, 2024-09-13T12:06:47.134 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_affinity": 1, 2024-09-13T12:06:47.134 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_begin": 5, 2024-09-13T12:06:47.134 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_end": 29, 2024-09-13T12:06:47.135 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 36, 2024-09-13T12:06:47.135 INFO:tasks.workunit.client.0.smithi032.stdout: "up_thru": 40, 2024-09-13T12:06:47.135 INFO:tasks.workunit.client.0.smithi032.stdout: "down_at": 30, 2024-09-13T12:06:47.135 INFO:tasks.workunit.client.0.smithi032.stdout: "lost_at": 0, 2024-09-13T12:06:47.135 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addrs": { 2024-09-13T12:06:47.135 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:06:47.135 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.135 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:06:47.135 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6802", 2024-09-13T12:06:47.136 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3817518293 2024-09-13T12:06:47.136 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.136 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.136 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:06:47.136 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6803", 2024-09-13T12:06:47.136 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3817518293 2024-09-13T12:06:47.136 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.136 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:47.136 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.136 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addrs": { 2024-09-13T12:06:47.136 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:06:47.136 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.137 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:06:47.137 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6804", 2024-09-13T12:06:47.137 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3817518293 2024-09-13T12:06:47.137 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.137 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.137 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:06:47.137 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6805", 2024-09-13T12:06:47.137 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3817518293 2024-09-13T12:06:47.137 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.137 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:47.137 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.137 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-13T12:06:47.137 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:06:47.137 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.138 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:06:47.138 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6808", 2024-09-13T12:06:47.138 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3817518293 2024-09-13T12:06:47.138 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.138 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.138 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:06:47.138 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6809", 2024-09-13T12:06:47.138 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3817518293 2024-09-13T12:06:47.138 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.138 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:47.138 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.138 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-13T12:06:47.138 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:06:47.139 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.139 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:06:47.139 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6806", 2024-09-13T12:06:47.139 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3817518293 2024-09-13T12:06:47.139 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.139 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.139 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:06:47.139 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6807", 2024-09-13T12:06:47.139 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3817518293 2024-09-13T12:06:47.139 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.139 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:47.139 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.139 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addr": "127.0.0.1:6803/3817518293", 2024-09-13T12:06:47.140 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addr": "127.0.0.1:6805/3817518293", 2024-09-13T12:06:47.140 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addr": "127.0.0.1:6809/3817518293", 2024-09-13T12:06:47.140 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addr": "127.0.0.1:6807/3817518293", 2024-09-13T12:06:47.140 INFO:tasks.workunit.client.0.smithi032.stdout: "state": [ 2024-09-13T12:06:47.140 INFO:tasks.workunit.client.0.smithi032.stdout: "exists", 2024-09-13T12:06:47.140 INFO:tasks.workunit.client.0.smithi032.stdout: "up" 2024-09-13T12:06:47.140 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:47.140 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.140 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.140 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 1, 2024-09-13T12:06:47.140 INFO:tasks.workunit.client.0.smithi032.stdout: "uuid": "eb913216-75c2-4cc2-ad3b-855fbbad0fc3", 2024-09-13T12:06:47.140 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 0, 2024-09-13T12:06:47.140 INFO:tasks.workunit.client.0.smithi032.stdout: "in": 1, 2024-09-13T12:06:47.141 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 1, 2024-09-13T12:06:47.141 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_affinity": 1, 2024-09-13T12:06:47.141 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_begin": 0, 2024-09-13T12:06:47.141 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_end": 0, 2024-09-13T12:06:47.141 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 10, 2024-09-13T12:06:47.141 INFO:tasks.workunit.client.0.smithi032.stdout: "up_thru": 23, 2024-09-13T12:06:47.141 INFO:tasks.workunit.client.0.smithi032.stdout: "down_at": 31, 2024-09-13T12:06:47.141 INFO:tasks.workunit.client.0.smithi032.stdout: "lost_at": 0, 2024-09-13T12:06:47.141 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addrs": { 2024-09-13T12:06:47.141 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:06:47.141 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.141 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:06:47.141 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6810", 2024-09-13T12:06:47.141 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1355014695 2024-09-13T12:06:47.142 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.142 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.142 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:06:47.142 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6811", 2024-09-13T12:06:47.142 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1355014695 2024-09-13T12:06:47.142 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.142 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:47.142 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.142 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addrs": { 2024-09-13T12:06:47.142 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:06:47.142 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.142 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:06:47.142 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6812", 2024-09-13T12:06:47.143 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1355014695 2024-09-13T12:06:47.143 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.143 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.143 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:06:47.143 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6813", 2024-09-13T12:06:47.143 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1355014695 2024-09-13T12:06:47.143 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.143 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:47.143 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.143 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-13T12:06:47.143 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:06:47.143 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.143 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:06:47.144 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6816", 2024-09-13T12:06:47.144 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1355014695 2024-09-13T12:06:47.144 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.144 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.144 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:06:47.144 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6817", 2024-09-13T12:06:47.144 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1355014695 2024-09-13T12:06:47.144 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.144 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:47.144 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.144 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-13T12:06:47.144 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:06:47.144 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.145 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:06:47.145 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6814", 2024-09-13T12:06:47.145 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1355014695 2024-09-13T12:06:47.145 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.145 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.145 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:06:47.145 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6815", 2024-09-13T12:06:47.145 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1355014695 2024-09-13T12:06:47.145 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.145 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:47.145 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.145 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addr": "127.0.0.1:6811/1355014695", 2024-09-13T12:06:47.145 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addr": "127.0.0.1:6813/1355014695", 2024-09-13T12:06:47.145 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1355014695", 2024-09-13T12:06:47.146 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1355014695", 2024-09-13T12:06:47.146 INFO:tasks.workunit.client.0.smithi032.stdout: "state": [ 2024-09-13T12:06:47.146 INFO:tasks.workunit.client.0.smithi032.stdout: "exists" 2024-09-13T12:06:47.146 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:47.146 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.146 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.146 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2, 2024-09-13T12:06:47.146 INFO:tasks.workunit.client.0.smithi032.stdout: "uuid": "69901422-110f-4bc1-8cb2-72564b47a69b", 2024-09-13T12:06:47.146 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 1, 2024-09-13T12:06:47.146 INFO:tasks.workunit.client.0.smithi032.stdout: "in": 1, 2024-09-13T12:06:47.146 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 1, 2024-09-13T12:06:47.146 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_affinity": 1, 2024-09-13T12:06:47.146 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_begin": 15, 2024-09-13T12:06:47.146 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_end": 29, 2024-09-13T12:06:47.147 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 38, 2024-09-13T12:06:47.147 INFO:tasks.workunit.client.0.smithi032.stdout: "up_thru": 0, 2024-09-13T12:06:47.147 INFO:tasks.workunit.client.0.smithi032.stdout: "down_at": 32, 2024-09-13T12:06:47.147 INFO:tasks.workunit.client.0.smithi032.stdout: "lost_at": 0, 2024-09-13T12:06:47.147 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addrs": { 2024-09-13T12:06:47.147 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:06:47.147 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.147 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:06:47.147 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6810", 2024-09-13T12:06:47.147 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 746434650 2024-09-13T12:06:47.147 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.147 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.148 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:06:47.148 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6811", 2024-09-13T12:06:47.148 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 746434650 2024-09-13T12:06:47.148 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.148 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:47.148 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.148 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addrs": { 2024-09-13T12:06:47.148 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:06:47.148 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.148 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:06:47.148 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6812", 2024-09-13T12:06:47.148 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 746434650 2024-09-13T12:06:47.148 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.148 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.149 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:06:47.149 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6813", 2024-09-13T12:06:47.149 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 746434650 2024-09-13T12:06:47.149 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.149 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:47.149 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.149 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-13T12:06:47.149 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:06:47.149 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.149 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:06:47.149 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6816", 2024-09-13T12:06:47.149 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 746434650 2024-09-13T12:06:47.149 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.149 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.150 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:06:47.150 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6817", 2024-09-13T12:06:47.150 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 746434650 2024-09-13T12:06:47.150 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.150 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:47.150 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.150 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-13T12:06:47.150 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:06:47.150 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.150 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:06:47.150 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6814", 2024-09-13T12:06:47.151 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 746434650 2024-09-13T12:06:47.151 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.151 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.151 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:06:47.151 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6815", 2024-09-13T12:06:47.151 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 746434650 2024-09-13T12:06:47.151 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.151 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:47.151 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.151 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addr": "127.0.0.1:6811/746434650", 2024-09-13T12:06:47.151 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addr": "127.0.0.1:6813/746434650", 2024-09-13T12:06:47.151 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addr": "127.0.0.1:6817/746434650", 2024-09-13T12:06:47.152 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addr": "127.0.0.1:6815/746434650", 2024-09-13T12:06:47.152 INFO:tasks.workunit.client.0.smithi032.stdout: "state": [ 2024-09-13T12:06:47.152 INFO:tasks.workunit.client.0.smithi032.stdout: "exists", 2024-09-13T12:06:47.152 INFO:tasks.workunit.client.0.smithi032.stdout: "up" 2024-09-13T12:06:47.152 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:47.152 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.152 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:47.152 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_xinfo": [ 2024-09-13T12:06:47.152 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.152 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 0, 2024-09-13T12:06:47.152 INFO:tasks.workunit.client.0.smithi032.stdout: "down_stamp": "2024-09-13T12:05:16.553008+0000", 2024-09-13T12:06:47.152 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_probability": 0, 2024-09-13T12:06:47.153 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_interval": 0, 2024-09-13T12:06:47.153 INFO:tasks.workunit.client.0.smithi032.stdout: "features": 4540719139924082687, 2024-09-13T12:06:47.153 INFO:tasks.workunit.client.0.smithi032.stdout: "old_weight": 0, 2024-09-13T12:06:47.153 INFO:tasks.workunit.client.0.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-13T12:03:15.350153+0000", 2024-09-13T12:06:47.153 INFO:tasks.workunit.client.0.smithi032.stdout: "dead_epoch": 0 2024-09-13T12:06:47.153 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.153 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.153 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 1, 2024-09-13T12:06:47.153 INFO:tasks.workunit.client.0.smithi032.stdout: "down_stamp": "2024-09-13T12:05:17.080406+0000", 2024-09-13T12:06:47.153 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_probability": 0, 2024-09-13T12:06:47.153 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_interval": 0, 2024-09-13T12:06:47.153 INFO:tasks.workunit.client.0.smithi032.stdout: "features": 4540719139924082687, 2024-09-13T12:06:47.154 INFO:tasks.workunit.client.0.smithi032.stdout: "old_weight": 0, 2024-09-13T12:06:47.154 INFO:tasks.workunit.client.0.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-13T12:03:51.563515+0000", 2024-09-13T12:06:47.154 INFO:tasks.workunit.client.0.smithi032.stdout: "dead_epoch": 0 2024-09-13T12:06:47.154 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.154 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.154 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2, 2024-09-13T12:06:47.154 INFO:tasks.workunit.client.0.smithi032.stdout: "down_stamp": "2024-09-13T12:05:17.687936+0000", 2024-09-13T12:06:47.154 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_probability": 0, 2024-09-13T12:06:47.154 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_interval": 0, 2024-09-13T12:06:47.154 INFO:tasks.workunit.client.0.smithi032.stdout: "features": 4540719139924082687, 2024-09-13T12:06:47.154 INFO:tasks.workunit.client.0.smithi032.stdout: "old_weight": 0, 2024-09-13T12:06:47.154 INFO:tasks.workunit.client.0.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-13T12:04:26.609710+0000", 2024-09-13T12:06:47.154 INFO:tasks.workunit.client.0.smithi032.stdout: "dead_epoch": 0 2024-09-13T12:06:47.155 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.155 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:47.155 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_upmap": [], 2024-09-13T12:06:47.155 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_upmap_items": [], 2024-09-13T12:06:47.155 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_upmap_primaries": [], 2024-09-13T12:06:47.155 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_temp": [], 2024-09-13T12:06:47.155 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_temp": [], 2024-09-13T12:06:47.155 INFO:tasks.workunit.client.0.smithi032.stdout: "blocklist": {}, 2024-09-13T12:06:47.155 INFO:tasks.workunit.client.0.smithi032.stdout: "range_blocklist": {}, 2024-09-13T12:06:47.155 INFO:tasks.workunit.client.0.smithi032.stdout: "erasure_code_profiles": { 2024-09-13T12:06:47.155 INFO:tasks.workunit.client.0.smithi032.stdout: "default": { 2024-09-13T12:06:47.155 INFO:tasks.workunit.client.0.smithi032.stdout: "k": "2", 2024-09-13T12:06:47.155 INFO:tasks.workunit.client.0.smithi032.stdout: "m": "2", 2024-09-13T12:06:47.155 INFO:tasks.workunit.client.0.smithi032.stdout: "plugin": "jerasure", 2024-09-13T12:06:47.156 INFO:tasks.workunit.client.0.smithi032.stdout: "technique": "reed_sol_van" 2024-09-13T12:06:47.156 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.156 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.156 INFO:tasks.workunit.client.0.smithi032.stdout: "removed_snaps_queue": [], 2024-09-13T12:06:47.156 INFO:tasks.workunit.client.0.smithi032.stdout: "new_removed_snaps": [], 2024-09-13T12:06:47.156 INFO:tasks.workunit.client.0.smithi032.stdout: "new_purged_snaps": [], 2024-09-13T12:06:47.156 INFO:tasks.workunit.client.0.smithi032.stdout: "crush_node_flags": {}, 2024-09-13T12:06:47.156 INFO:tasks.workunit.client.0.smithi032.stdout: "device_class_flags": {}, 2024-09-13T12:06:47.156 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_mode": { 2024-09-13T12:06:47.156 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_mode_enabled": false, 2024-09-13T12:06:47.156 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_bucket_count": 0, 2024-09-13T12:06:47.156 INFO:tasks.workunit.client.0.smithi032.stdout: "degraded_stretch_mode": 0, 2024-09-13T12:06:47.157 INFO:tasks.workunit.client.0.smithi032.stdout: "recovering_stretch_mode": 0, 2024-09-13T12:06:47.157 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_mode_bucket": 0 2024-09-13T12:06:47.157 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.157 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.157 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_metadata": [ 2024-09-13T12:06:47.157 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.157 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:06:47.157 INFO:tasks.workunit.client.0.smithi032.stdout: "arch": "x86_64", 2024-09-13T12:06:47.157 INFO:tasks.workunit.client.0.smithi032.stdout: "back_addr": "[v2:127.0.0.1:6804/3817518293,v1:127.0.0.1:6805/3817518293]", 2024-09-13T12:06:47.157 INFO:tasks.workunit.client.0.smithi032.stdout: "back_iface": "", 2024-09-13T12:06:47.157 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs": "1", 2024-09-13T12:06:47.157 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_db": "0", 2024-09-13T12:06:47.158 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_wal": "0", 2024-09-13T12:06:47.158 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_single_shared_device": "1", 2024-09-13T12:06:47.158 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_allocation_from_file": "0", 2024-09-13T12:06:47.158 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_access_mode": "file", 2024-09-13T12:06:47.158 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_block_size": "4096", 2024-09-13T12:06:47.158 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_devices": "sda", 2024-09-13T12:06:47.158 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-13T12:06:47.158 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-13T12:06:47.158 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-13T12:06:47.158 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_rotational": "1", 2024-09-13T12:06:47.158 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_size": "107374182400", 2024-09-13T12:06:47.158 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_support_discard": "1", 2024-09-13T12:06:47.158 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_type": "hdd", 2024-09-13T12:06:47.158 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_min_alloc_size": "4096", 2024-09-13T12:06:47.159 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_release": "squid", 2024-09-13T12:06:47.159 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:06:47.159 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_short": "19.3.0-4697-gd58f7306", 2024-09-13T12:06:47.159 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_when_created": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:06:47.159 INFO:tasks.workunit.client.0.smithi032.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-13T12:06:47.159 INFO:tasks.workunit.client.0.smithi032.stdout: "created_at": "2024-09-13T12:02:48.448877Z", 2024-09-13T12:06:47.159 INFO:tasks.workunit.client.0.smithi032.stdout: "default_device_class": "hdd", 2024-09-13T12:06:47.159 INFO:tasks.workunit.client.0.smithi032.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4J2LL", 2024-09-13T12:06:47.159 INFO:tasks.workunit.client.0.smithi032.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-13T12:06:47.159 INFO:tasks.workunit.client.0.smithi032.stdout: "devices": "sda", 2024-09-13T12:06:47.159 INFO:tasks.workunit.client.0.smithi032.stdout: "distro": "centos", 2024-09-13T12:06:47.159 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_description": "CentOS Stream 9", 2024-09-13T12:06:47.160 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_version": "9", 2024-09-13T12:06:47.160 INFO:tasks.workunit.client.0.smithi032.stdout: "front_addr": "[v2:127.0.0.1:6802/3817518293,v1:127.0.0.1:6803/3817518293]", 2024-09-13T12:06:47.160 INFO:tasks.workunit.client.0.smithi032.stdout: "front_iface": "", 2024-09-13T12:06:47.160 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/3817518293,v1:127.0.0.1:6809/3817518293]", 2024-09-13T12:06:47.160 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/3817518293,v1:127.0.0.1:6807/3817518293]", 2024-09-13T12:06:47.160 INFO:tasks.workunit.client.0.smithi032.stdout: "hostname": "smithi032", 2024-09-13T12:06:47.160 INFO:tasks.workunit.client.0.smithi032.stdout: "journal_rotational": "1", 2024-09-13T12:06:47.160 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-13T12:06:47.160 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-13T12:06:47.160 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_swap_kb": "0", 2024-09-13T12:06:47.160 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_total_kb": "32493484", 2024-09-13T12:06:47.161 INFO:tasks.workunit.client.0.smithi032.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-13T12:06:47.161 INFO:tasks.workunit.client.0.smithi032.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-13T12:06:47.161 INFO:tasks.workunit.client.0.smithi032.stdout: "os": "Linux", 2024-09-13T12:06:47.161 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_data": "td/divergent-priors/0", 2024-09-13T12:06:47.161 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_objectstore": "bluestore", 2024-09-13T12:06:47.161 INFO:tasks.workunit.client.0.smithi032.stdout: "osdspec_affinity": "", 2024-09-13T12:06:47.161 INFO:tasks.workunit.client.0.smithi032.stdout: "rotational": "1" 2024-09-13T12:06:47.161 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.161 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.161 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 1, 2024-09-13T12:06:47.161 INFO:tasks.workunit.client.0.smithi032.stdout: "arch": "x86_64", 2024-09-13T12:06:47.161 INFO:tasks.workunit.client.0.smithi032.stdout: "back_addr": "[v2:127.0.0.1:6812/1355014695,v1:127.0.0.1:6813/1355014695]", 2024-09-13T12:06:47.161 INFO:tasks.workunit.client.0.smithi032.stdout: "back_iface": "", 2024-09-13T12:06:47.162 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs": "1", 2024-09-13T12:06:47.162 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_db": "0", 2024-09-13T12:06:47.162 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_wal": "0", 2024-09-13T12:06:47.162 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_single_shared_device": "1", 2024-09-13T12:06:47.162 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_allocation_from_file": "0", 2024-09-13T12:06:47.162 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_access_mode": "file", 2024-09-13T12:06:47.162 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_block_size": "4096", 2024-09-13T12:06:47.162 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_devices": "sda", 2024-09-13T12:06:47.162 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-13T12:06:47.162 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-13T12:06:47.162 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-13T12:06:47.162 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_rotational": "1", 2024-09-13T12:06:47.162 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_size": "107374182400", 2024-09-13T12:06:47.162 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_support_discard": "1", 2024-09-13T12:06:47.163 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_type": "hdd", 2024-09-13T12:06:47.163 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_min_alloc_size": "4096", 2024-09-13T12:06:47.163 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_release": "squid", 2024-09-13T12:06:47.163 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:06:47.163 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_short": "19.3.0-4697-gd58f7306", 2024-09-13T12:06:47.163 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_when_created": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:06:47.163 INFO:tasks.workunit.client.0.smithi032.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-13T12:06:47.163 INFO:tasks.workunit.client.0.smithi032.stdout: "created_at": "2024-09-13T12:03:23.158100Z", 2024-09-13T12:06:47.163 INFO:tasks.workunit.client.0.smithi032.stdout: "default_device_class": "hdd", 2024-09-13T12:06:47.163 INFO:tasks.workunit.client.0.smithi032.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4J2LL", 2024-09-13T12:06:47.163 INFO:tasks.workunit.client.0.smithi032.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-13T12:06:47.163 INFO:tasks.workunit.client.0.smithi032.stdout: "devices": "sda", 2024-09-13T12:06:47.163 INFO:tasks.workunit.client.0.smithi032.stdout: "distro": "centos", 2024-09-13T12:06:47.163 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_description": "CentOS Stream 9", 2024-09-13T12:06:47.164 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_version": "9", 2024-09-13T12:06:47.164 INFO:tasks.workunit.client.0.smithi032.stdout: "front_addr": "[v2:127.0.0.1:6810/1355014695,v1:127.0.0.1:6811/1355014695]", 2024-09-13T12:06:47.164 INFO:tasks.workunit.client.0.smithi032.stdout: "front_iface": "", 2024-09-13T12:06:47.164 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1355014695,v1:127.0.0.1:6817/1355014695]", 2024-09-13T12:06:47.164 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1355014695,v1:127.0.0.1:6815/1355014695]", 2024-09-13T12:06:47.164 INFO:tasks.workunit.client.0.smithi032.stdout: "hostname": "smithi032", 2024-09-13T12:06:47.164 INFO:tasks.workunit.client.0.smithi032.stdout: "journal_rotational": "1", 2024-09-13T12:06:47.164 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-13T12:06:47.164 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-13T12:06:47.164 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_swap_kb": "0", 2024-09-13T12:06:47.164 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_total_kb": "32493484", 2024-09-13T12:06:47.164 INFO:tasks.workunit.client.0.smithi032.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-13T12:06:47.164 INFO:tasks.workunit.client.0.smithi032.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-13T12:06:47.164 INFO:tasks.workunit.client.0.smithi032.stdout: "os": "Linux", 2024-09-13T12:06:47.165 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_data": "td/divergent-priors/1", 2024-09-13T12:06:47.165 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_objectstore": "bluestore", 2024-09-13T12:06:47.165 INFO:tasks.workunit.client.0.smithi032.stdout: "osdspec_affinity": "", 2024-09-13T12:06:47.165 INFO:tasks.workunit.client.0.smithi032.stdout: "rotational": "1" 2024-09-13T12:06:47.165 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.165 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.165 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:06:47.165 INFO:tasks.workunit.client.0.smithi032.stdout: "arch": "x86_64", 2024-09-13T12:06:47.165 INFO:tasks.workunit.client.0.smithi032.stdout: "back_addr": "[v2:127.0.0.1:6812/746434650,v1:127.0.0.1:6813/746434650]", 2024-09-13T12:06:47.165 INFO:tasks.workunit.client.0.smithi032.stdout: "back_iface": "", 2024-09-13T12:06:47.165 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs": "1", 2024-09-13T12:06:47.165 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_db": "0", 2024-09-13T12:06:47.165 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_wal": "0", 2024-09-13T12:06:47.165 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_single_shared_device": "1", 2024-09-13T12:06:47.166 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_allocation_from_file": "0", 2024-09-13T12:06:47.166 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_access_mode": "file", 2024-09-13T12:06:47.166 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_block_size": "4096", 2024-09-13T12:06:47.166 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_devices": "sda", 2024-09-13T12:06:47.166 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-13T12:06:47.166 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-13T12:06:47.166 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-13T12:06:47.166 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_rotational": "1", 2024-09-13T12:06:47.166 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_size": "107374182400", 2024-09-13T12:06:47.166 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_support_discard": "1", 2024-09-13T12:06:47.166 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_type": "hdd", 2024-09-13T12:06:47.166 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_min_alloc_size": "4096", 2024-09-13T12:06:47.166 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_release": "squid", 2024-09-13T12:06:47.167 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:06:47.167 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_short": "19.3.0-4697-gd58f7306", 2024-09-13T12:06:47.167 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_when_created": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:06:47.167 INFO:tasks.workunit.client.0.smithi032.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-13T12:06:47.167 INFO:tasks.workunit.client.0.smithi032.stdout: "created_at": "2024-09-13T12:03:58.721983Z", 2024-09-13T12:06:47.167 INFO:tasks.workunit.client.0.smithi032.stdout: "default_device_class": "hdd", 2024-09-13T12:06:47.167 INFO:tasks.workunit.client.0.smithi032.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4J2LL", 2024-09-13T12:06:47.167 INFO:tasks.workunit.client.0.smithi032.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-13T12:06:47.167 INFO:tasks.workunit.client.0.smithi032.stdout: "devices": "sda", 2024-09-13T12:06:47.167 INFO:tasks.workunit.client.0.smithi032.stdout: "distro": "centos", 2024-09-13T12:06:47.167 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_description": "CentOS Stream 9", 2024-09-13T12:06:47.167 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_version": "9", 2024-09-13T12:06:47.167 INFO:tasks.workunit.client.0.smithi032.stdout: "front_addr": "[v2:127.0.0.1:6810/746434650,v1:127.0.0.1:6811/746434650]", 2024-09-13T12:06:47.167 INFO:tasks.workunit.client.0.smithi032.stdout: "front_iface": "", 2024-09-13T12:06:47.168 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/746434650,v1:127.0.0.1:6817/746434650]", 2024-09-13T12:06:47.168 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/746434650,v1:127.0.0.1:6815/746434650]", 2024-09-13T12:06:47.168 INFO:tasks.workunit.client.0.smithi032.stdout: "hostname": "smithi032", 2024-09-13T12:06:47.168 INFO:tasks.workunit.client.0.smithi032.stdout: "journal_rotational": "1", 2024-09-13T12:06:47.168 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-13T12:06:47.168 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-13T12:06:47.168 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_swap_kb": "0", 2024-09-13T12:06:47.168 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_total_kb": "32493484", 2024-09-13T12:06:47.168 INFO:tasks.workunit.client.0.smithi032.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-13T12:06:47.168 INFO:tasks.workunit.client.0.smithi032.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-13T12:06:47.168 INFO:tasks.workunit.client.0.smithi032.stdout: "os": "Linux", 2024-09-13T12:06:47.168 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_data": "td/divergent-priors/2", 2024-09-13T12:06:47.168 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_objectstore": "bluestore", 2024-09-13T12:06:47.169 INFO:tasks.workunit.client.0.smithi032.stdout: "osdspec_affinity": "", 2024-09-13T12:06:47.169 INFO:tasks.workunit.client.0.smithi032.stdout: "rotational": "1" 2024-09-13T12:06:47.169 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.169 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:47.169 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap_clean_epochs": { 2024-09-13T12:06:47.169 INFO:tasks.workunit.client.0.smithi032.stdout: "min_last_epoch_clean": 0, 2024-09-13T12:06:47.169 INFO:tasks.workunit.client.0.smithi032.stdout: "last_epoch_clean": { 2024-09-13T12:06:47.169 INFO:tasks.workunit.client.0.smithi032.stdout: "per_pool": [] 2024-09-13T12:06:47.169 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.169 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_epochs": [ 2024-09-13T12:06:47.169 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.169 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:06:47.169 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 36 2024-09-13T12:06:47.169 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.170 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.170 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:06:47.170 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 38 2024-09-13T12:06:47.170 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.170 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:47.170 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.170 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap_first_committed": 1, 2024-09-13T12:06:47.170 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap_last_committed": 41, 2024-09-13T12:06:47.170 INFO:tasks.workunit.client.0.smithi032.stdout: "crushmap": { 2024-09-13T12:06:47.170 INFO:tasks.workunit.client.0.smithi032.stdout: "devices": [ 2024-09-13T12:06:47.170 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.170 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:06:47.170 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "osd.0", 2024-09-13T12:06:47.171 INFO:tasks.workunit.client.0.smithi032.stdout: "class": "hdd" 2024-09-13T12:06:47.171 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.171 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.171 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 1, 2024-09-13T12:06:47.171 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "osd.1", 2024-09-13T12:06:47.171 INFO:tasks.workunit.client.0.smithi032.stdout: "class": "hdd" 2024-09-13T12:06:47.171 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.171 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.171 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:06:47.171 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "osd.2", 2024-09-13T12:06:47.171 INFO:tasks.workunit.client.0.smithi032.stdout: "class": "hdd" 2024-09-13T12:06:47.171 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.171 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:47.172 INFO:tasks.workunit.client.0.smithi032.stdout: "types": [ 2024-09-13T12:06:47.172 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.172 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 0, 2024-09-13T12:06:47.172 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "osd" 2024-09-13T12:06:47.172 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.172 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.172 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 1, 2024-09-13T12:06:47.172 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "host" 2024-09-13T12:06:47.172 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.172 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.172 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 2, 2024-09-13T12:06:47.172 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "chassis" 2024-09-13T12:06:47.172 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.173 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.173 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 3, 2024-09-13T12:06:47.173 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "rack" 2024-09-13T12:06:47.173 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.173 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.173 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 4, 2024-09-13T12:06:47.173 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "row" 2024-09-13T12:06:47.173 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.173 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.173 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 5, 2024-09-13T12:06:47.173 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "pdu" 2024-09-13T12:06:47.173 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.173 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.173 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 6, 2024-09-13T12:06:47.174 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "pod" 2024-09-13T12:06:47.174 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.174 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.174 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 7, 2024-09-13T12:06:47.174 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "room" 2024-09-13T12:06:47.174 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.174 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.174 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 8, 2024-09-13T12:06:47.174 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "datacenter" 2024-09-13T12:06:47.174 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.174 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.174 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 9, 2024-09-13T12:06:47.174 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "zone" 2024-09-13T12:06:47.175 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.175 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.175 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 10, 2024-09-13T12:06:47.175 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "region" 2024-09-13T12:06:47.175 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.175 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.175 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 11, 2024-09-13T12:06:47.175 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "root" 2024-09-13T12:06:47.175 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.175 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:47.175 INFO:tasks.workunit.client.0.smithi032.stdout: "buckets": [ 2024-09-13T12:06:47.175 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.175 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -1, 2024-09-13T12:06:47.175 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "default", 2024-09-13T12:06:47.176 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 11, 2024-09-13T12:06:47.176 INFO:tasks.workunit.client.0.smithi032.stdout: "type_name": "root", 2024-09-13T12:06:47.176 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:06:47.176 INFO:tasks.workunit.client.0.smithi032.stdout: "alg": "straw2", 2024-09-13T12:06:47.176 INFO:tasks.workunit.client.0.smithi032.stdout: "hash": "rjenkins1", 2024-09-13T12:06:47.176 INFO:tasks.workunit.client.0.smithi032.stdout: "items": [ 2024-09-13T12:06:47.176 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.176 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -3, 2024-09-13T12:06:47.176 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:06:47.176 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 0 2024-09-13T12:06:47.176 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.176 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:47.176 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.176 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.177 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -2, 2024-09-13T12:06:47.177 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "default~hdd", 2024-09-13T12:06:47.177 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 11, 2024-09-13T12:06:47.177 INFO:tasks.workunit.client.0.smithi032.stdout: "type_name": "root", 2024-09-13T12:06:47.177 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:06:47.177 INFO:tasks.workunit.client.0.smithi032.stdout: "alg": "straw2", 2024-09-13T12:06:47.177 INFO:tasks.workunit.client.0.smithi032.stdout: "hash": "rjenkins1", 2024-09-13T12:06:47.177 INFO:tasks.workunit.client.0.smithi032.stdout: "items": [ 2024-09-13T12:06:47.177 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.177 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -4, 2024-09-13T12:06:47.177 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:06:47.177 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 0 2024-09-13T12:06:47.177 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.177 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:47.178 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.178 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.178 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -3, 2024-09-13T12:06:47.178 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "smithi032", 2024-09-13T12:06:47.178 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 1, 2024-09-13T12:06:47.178 INFO:tasks.workunit.client.0.smithi032.stdout: "type_name": "host", 2024-09-13T12:06:47.178 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:06:47.178 INFO:tasks.workunit.client.0.smithi032.stdout: "alg": "straw2", 2024-09-13T12:06:47.178 INFO:tasks.workunit.client.0.smithi032.stdout: "hash": "rjenkins1", 2024-09-13T12:06:47.178 INFO:tasks.workunit.client.0.smithi032.stdout: "items": [ 2024-09-13T12:06:47.178 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.178 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:06:47.178 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:06:47.179 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 0 2024-09-13T12:06:47.179 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.179 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.179 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 1, 2024-09-13T12:06:47.179 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:06:47.179 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 1 2024-09-13T12:06:47.179 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.179 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.179 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:06:47.179 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:06:47.179 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 2 2024-09-13T12:06:47.179 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.179 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:47.179 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.180 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.180 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -4, 2024-09-13T12:06:47.180 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "smithi032~hdd", 2024-09-13T12:06:47.180 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 1, 2024-09-13T12:06:47.180 INFO:tasks.workunit.client.0.smithi032.stdout: "type_name": "host", 2024-09-13T12:06:47.180 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:06:47.180 INFO:tasks.workunit.client.0.smithi032.stdout: "alg": "straw2", 2024-09-13T12:06:47.180 INFO:tasks.workunit.client.0.smithi032.stdout: "hash": "rjenkins1", 2024-09-13T12:06:47.180 INFO:tasks.workunit.client.0.smithi032.stdout: "items": [ 2024-09-13T12:06:47.180 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.180 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:06:47.180 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:06:47.180 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 0 2024-09-13T12:06:47.181 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.181 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.181 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 1, 2024-09-13T12:06:47.181 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:06:47.181 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 1 2024-09-13T12:06:47.181 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.181 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.181 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:06:47.181 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:06:47.181 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 2 2024-09-13T12:06:47.181 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.181 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:47.181 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.181 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:47.182 INFO:tasks.workunit.client.0.smithi032.stdout: "rules": [ 2024-09-13T12:06:47.182 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.182 INFO:tasks.workunit.client.0.smithi032.stdout: "rule_id": 0, 2024-09-13T12:06:47.182 INFO:tasks.workunit.client.0.smithi032.stdout: "rule_name": "replicated_rule", 2024-09-13T12:06:47.182 INFO:tasks.workunit.client.0.smithi032.stdout: "type": 1, 2024-09-13T12:06:47.182 INFO:tasks.workunit.client.0.smithi032.stdout: "steps": [ 2024-09-13T12:06:47.182 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.182 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "take", 2024-09-13T12:06:47.182 INFO:tasks.workunit.client.0.smithi032.stdout: "item": -1, 2024-09-13T12:06:47.182 INFO:tasks.workunit.client.0.smithi032.stdout: "item_name": "default" 2024-09-13T12:06:47.182 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.182 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.182 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "choose_firstn", 2024-09-13T12:06:47.183 INFO:tasks.workunit.client.0.smithi032.stdout: "num": 0, 2024-09-13T12:06:47.183 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "osd" 2024-09-13T12:06:47.183 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.183 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.183 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "emit" 2024-09-13T12:06:47.183 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.183 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:06:47.183 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.183 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:47.183 INFO:tasks.workunit.client.0.smithi032.stdout: "tunables": { 2024-09-13T12:06:47.183 INFO:tasks.workunit.client.0.smithi032.stdout: "choose_local_tries": 0, 2024-09-13T12:06:47.183 INFO:tasks.workunit.client.0.smithi032.stdout: "choose_local_fallback_tries": 0, 2024-09-13T12:06:47.183 INFO:tasks.workunit.client.0.smithi032.stdout: "choose_total_tries": 50, 2024-09-13T12:06:47.184 INFO:tasks.workunit.client.0.smithi032.stdout: "chooseleaf_descend_once": 1, 2024-09-13T12:06:47.184 INFO:tasks.workunit.client.0.smithi032.stdout: "chooseleaf_vary_r": 1, 2024-09-13T12:06:47.184 INFO:tasks.workunit.client.0.smithi032.stdout: "chooseleaf_stable": 1, 2024-09-13T12:06:47.184 INFO:tasks.workunit.client.0.smithi032.stdout: "msr_descents": 100, 2024-09-13T12:06:47.184 INFO:tasks.workunit.client.0.smithi032.stdout: "msr_collision_tries": 100, 2024-09-13T12:06:47.184 INFO:tasks.workunit.client.0.smithi032.stdout: "straw_calc_version": 1, 2024-09-13T12:06:47.184 INFO:tasks.workunit.client.0.smithi032.stdout: "allowed_bucket_algs": 54, 2024-09-13T12:06:47.184 INFO:tasks.workunit.client.0.smithi032.stdout: "profile": "jewel", 2024-09-13T12:06:47.184 INFO:tasks.workunit.client.0.smithi032.stdout: "optimal_tunables": 1, 2024-09-13T12:06:47.184 INFO:tasks.workunit.client.0.smithi032.stdout: "legacy_tunables": 0, 2024-09-13T12:06:47.184 INFO:tasks.workunit.client.0.smithi032.stdout: "minimum_required_version": "jewel", 2024-09-13T12:06:47.184 INFO:tasks.workunit.client.0.smithi032.stdout: "require_feature_tunables": 1, 2024-09-13T12:06:47.184 INFO:tasks.workunit.client.0.smithi032.stdout: "require_feature_tunables2": 1, 2024-09-13T12:06:47.185 INFO:tasks.workunit.client.0.smithi032.stdout: "has_v2_rules": 0, 2024-09-13T12:06:47.185 INFO:tasks.workunit.client.0.smithi032.stdout: "require_feature_tunables3": 1, 2024-09-13T12:06:47.185 INFO:tasks.workunit.client.0.smithi032.stdout: "has_v3_rules": 0, 2024-09-13T12:06:47.185 INFO:tasks.workunit.client.0.smithi032.stdout: "has_v4_buckets": 1, 2024-09-13T12:06:47.185 INFO:tasks.workunit.client.0.smithi032.stdout: "require_feature_tunables5": 1, 2024-09-13T12:06:47.185 INFO:tasks.workunit.client.0.smithi032.stdout: "has_v5_rules": 0, 2024-09-13T12:06:47.185 INFO:tasks.workunit.client.0.smithi032.stdout: "has_msr_rules": 0 2024-09-13T12:06:47.185 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.185 INFO:tasks.workunit.client.0.smithi032.stdout: "choose_args": {} 2024-09-13T12:06:47.185 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.185 INFO:tasks.workunit.client.0.smithi032.stdout: "fsmap": { 2024-09-13T12:06:47.185 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 1, 2024-09-13T12:06:47.186 INFO:tasks.workunit.client.0.smithi032.stderr:report 841827131 2024-09-13T12:06:47.186 INFO:tasks.workunit.client.0.smithi032.stdout: "btime": "2024-09-13T12:02:42:144183+0000", 2024-09-13T12:06:47.186 INFO:tasks.workunit.client.0.smithi032.stdout: "default_fscid": -1, 2024-09-13T12:06:47.186 INFO:tasks.workunit.client.0.smithi032.stdout: "compat": { 2024-09-13T12:06:47.186 INFO:tasks.workunit.client.0.smithi032.stdout: "compat": {}, 2024-09-13T12:06:47.186 INFO:tasks.workunit.client.0.smithi032.stdout: "ro_compat": {}, 2024-09-13T12:06:47.187 INFO:tasks.workunit.client.0.smithi032.stdout: "incompat": { 2024-09-13T12:06:47.187 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_1": "base v0.20", 2024-09-13T12:06:47.187 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_2": "client writeable ranges", 2024-09-13T12:06:47.187 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_3": "default file layouts on dirs", 2024-09-13T12:06:47.187 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_4": "dir inode in separate object", 2024-09-13T12:06:47.187 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_5": "mds uses versioned encoding", 2024-09-13T12:06:47.187 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-13T12:06:47.187 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_8": "no anchor table", 2024-09-13T12:06:47.188 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_9": "file layout v2", 2024-09-13T12:06:47.188 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_10": "snaprealm v2", 2024-09-13T12:06:47.188 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_11": "minor log segments", 2024-09-13T12:06:47.188 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_12": "quiesce subvolumes" 2024-09-13T12:06:47.188 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.188 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.188 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_flags": { 2024-09-13T12:06:47.188 INFO:tasks.workunit.client.0.smithi032.stdout: "enable_multiple": true, 2024-09-13T12:06:47.188 INFO:tasks.workunit.client.0.smithi032.stdout: "ever_enabled_multiple": true 2024-09-13T12:06:47.188 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.188 INFO:tasks.workunit.client.0.smithi032.stdout: "standbys": [], 2024-09-13T12:06:47.189 INFO:tasks.workunit.client.0.smithi032.stdout: "filesystems": [] 2024-09-13T12:06:47.189 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.189 INFO:tasks.workunit.client.0.smithi032.stdout: "mdsmap_first_committed": 1, 2024-09-13T12:06:47.189 INFO:tasks.workunit.client.0.smithi032.stdout: "mdsmap_last_committed": 1, 2024-09-13T12:06:47.189 INFO:tasks.workunit.client.0.smithi032.stdout: "auth": { 2024-09-13T12:06:47.189 INFO:tasks.workunit.client.0.smithi032.stdout: "first_committed": 1, 2024-09-13T12:06:47.189 INFO:tasks.workunit.client.0.smithi032.stdout: "last_committed": 5, 2024-09-13T12:06:47.189 INFO:tasks.workunit.client.0.smithi032.stdout: "num_secrets": 3 2024-09-13T12:06:47.189 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.189 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg": 1, 2024-09-13T12:06:47.189 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg_active": 1, 2024-09-13T12:06:47.189 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg_unknown": 0, 2024-09-13T12:06:47.190 INFO:tasks.workunit.client.0.smithi032.stdout: "num_osd": 3, 2024-09-13T12:06:47.190 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_sum": { 2024-09-13T12:06:47.190 INFO:tasks.workunit.client.0.smithi032.stdout: "stat_sum": { 2024-09-13T12:06:47.190 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes": 47971, 2024-09-13T12:06:47.190 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects": 100, 2024-09-13T12:06:47.190 INFO:tasks.workunit.client.0.smithi032.stdout: "num_object_clones": 0, 2024-09-13T12:06:47.190 INFO:tasks.workunit.client.0.smithi032.stdout: "num_object_copies": 300, 2024-09-13T12:06:47.190 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_missing_on_primary": 0, 2024-09-13T12:06:47.190 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_missing": 0, 2024-09-13T12:06:47.190 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_degraded": 100, 2024-09-13T12:06:47.190 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_misplaced": 0, 2024-09-13T12:06:47.190 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_unfound": 0, 2024-09-13T12:06:47.190 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_dirty": 100, 2024-09-13T12:06:47.191 INFO:tasks.workunit.client.0.smithi032.stdout: "num_whiteouts": 0, 2024-09-13T12:06:47.191 INFO:tasks.workunit.client.0.smithi032.stdout: "num_read": 0, 2024-09-13T12:06:47.191 INFO:tasks.workunit.client.0.smithi032.stdout: "num_read_kb": 0, 2024-09-13T12:06:47.191 INFO:tasks.workunit.client.0.smithi032.stdout: "num_write": 101, 2024-09-13T12:06:47.191 INFO:tasks.workunit.client.0.smithi032.stdout: "num_write_kb": 101, 2024-09-13T12:06:47.191 INFO:tasks.workunit.client.0.smithi032.stdout: "num_scrub_errors": 0, 2024-09-13T12:06:47.191 INFO:tasks.workunit.client.0.smithi032.stdout: "num_shallow_scrub_errors": 0, 2024-09-13T12:06:47.191 INFO:tasks.workunit.client.0.smithi032.stdout: "num_deep_scrub_errors": 0, 2024-09-13T12:06:47.191 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_recovered": 0, 2024-09-13T12:06:47.191 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes_recovered": 0, 2024-09-13T12:06:47.191 INFO:tasks.workunit.client.0.smithi032.stdout: "num_keys_recovered": 0, 2024-09-13T12:06:47.191 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_omap": 0, 2024-09-13T12:06:47.191 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_hit_set_archive": 0, 2024-09-13T12:06:47.192 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes_hit_set_archive": 0, 2024-09-13T12:06:47.192 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush": 0, 2024-09-13T12:06:47.192 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_kb": 0, 2024-09-13T12:06:47.192 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict": 0, 2024-09-13T12:06:47.192 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_kb": 0, 2024-09-13T12:06:47.192 INFO:tasks.workunit.client.0.smithi032.stdout: "num_promote": 0, 2024-09-13T12:06:47.192 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_mode_high": 0, 2024-09-13T12:06:47.192 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_mode_low": 0, 2024-09-13T12:06:47.192 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_mode_some": 0, 2024-09-13T12:06:47.192 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_mode_full": 0, 2024-09-13T12:06:47.192 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_pinned": 0, 2024-09-13T12:06:47.192 INFO:tasks.workunit.client.0.smithi032.stdout: "num_legacy_snapsets": 0, 2024-09-13T12:06:47.193 INFO:tasks.workunit.client.0.smithi032.stdout: "num_large_omap_objects": 0, 2024-09-13T12:06:47.193 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_manifest": 0, 2024-09-13T12:06:47.193 INFO:tasks.workunit.client.0.smithi032.stdout: "num_omap_bytes": 0, 2024-09-13T12:06:47.193 INFO:tasks.workunit.client.0.smithi032.stdout: "num_omap_keys": 0, 2024-09-13T12:06:47.193 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_repaired": 0 2024-09-13T12:06:47.193 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.193 INFO:tasks.workunit.client.0.smithi032.stdout: "store_stats": { 2024-09-13T12:06:47.193 INFO:tasks.workunit.client.0.smithi032.stdout: "total": 0, 2024-09-13T12:06:47.193 INFO:tasks.workunit.client.0.smithi032.stdout: "available": 0, 2024-09-13T12:06:47.193 INFO:tasks.workunit.client.0.smithi032.stdout: "internally_reserved": 0, 2024-09-13T12:06:47.193 INFO:tasks.workunit.client.0.smithi032.stdout: "allocated": 0, 2024-09-13T12:06:47.193 INFO:tasks.workunit.client.0.smithi032.stdout: "data_stored": 0, 2024-09-13T12:06:47.194 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed": 0, 2024-09-13T12:06:47.194 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_allocated": 0, 2024-09-13T12:06:47.194 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_original": 0, 2024-09-13T12:06:47.194 INFO:tasks.workunit.client.0.smithi032.stdout: "omap_allocated": 0, 2024-09-13T12:06:47.194 INFO:tasks.workunit.client.0.smithi032.stdout: "internal_metadata": 0 2024-09-13T12:06:47.194 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.194 INFO:tasks.workunit.client.0.smithi032.stdout: "log_size": 17, 2024-09-13T12:06:47.194 INFO:tasks.workunit.client.0.smithi032.stdout: "ondisk_log_size": 17, 2024-09-13T12:06:47.194 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 2, 2024-09-13T12:06:47.194 INFO:tasks.workunit.client.0.smithi032.stdout: "acting": 2, 2024-09-13T12:06:47.194 INFO:tasks.workunit.client.0.smithi032.stdout: "num_store_stats": 0 2024-09-13T12:06:47.194 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.194 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_sum": { 2024-09-13T12:06:47.195 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 0, 2024-09-13T12:06:47.195 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 0, 2024-09-13T12:06:47.195 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pgs": 3, 2024-09-13T12:06:47.195 INFO:tasks.workunit.client.0.smithi032.stdout: "num_osds": 3, 2024-09-13T12:06:47.195 INFO:tasks.workunit.client.0.smithi032.stdout: "num_per_pool_osds": 3, 2024-09-13T12:06:47.195 INFO:tasks.workunit.client.0.smithi032.stdout: "num_per_pool_omap_osds": 3, 2024-09-13T12:06:47.195 INFO:tasks.workunit.client.0.smithi032.stdout: "kb": 314572800, 2024-09-13T12:06:47.195 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used": 83644, 2024-09-13T12:06:47.195 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_data": 2104, 2024-09-13T12:06:47.195 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_omap": 10, 2024-09-13T12:06:47.195 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_meta": 81461, 2024-09-13T12:06:47.195 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_avail": 314489156, 2024-09-13T12:06:47.195 INFO:tasks.workunit.client.0.smithi032.stdout: "statfs": { 2024-09-13T12:06:47.196 INFO:tasks.workunit.client.0.smithi032.stdout: "total": 322122547200, 2024-09-13T12:06:47.196 INFO:tasks.workunit.client.0.smithi032.stdout: "available": 322036895744, 2024-09-13T12:06:47.196 INFO:tasks.workunit.client.0.smithi032.stdout: "internally_reserved": 0, 2024-09-13T12:06:47.196 INFO:tasks.workunit.client.0.smithi032.stdout: "allocated": 2154496, 2024-09-13T12:06:47.196 INFO:tasks.workunit.client.0.smithi032.stdout: "data_stored": 417979, 2024-09-13T12:06:47.196 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed": 0, 2024-09-13T12:06:47.196 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_allocated": 0, 2024-09-13T12:06:47.196 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_original": 0, 2024-09-13T12:06:47.196 INFO:tasks.workunit.client.0.smithi032.stdout: "omap_allocated": 10397, 2024-09-13T12:06:47.196 INFO:tasks.workunit.client.0.smithi032.stdout: "internal_metadata": 83416931 2024-09-13T12:06:47.196 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.197 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_peers": [], 2024-09-13T12:06:47.197 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_trim_queue_len": 0, 2024-09-13T12:06:47.197 INFO:tasks.workunit.client.0.smithi032.stdout: "num_snap_trimming": 0, 2024-09-13T12:06:47.197 INFO:tasks.workunit.client.0.smithi032.stdout: "num_shards_repaired": 0, 2024-09-13T12:06:47.197 INFO:tasks.workunit.client.0.smithi032.stdout: "op_queue_age_hist": { 2024-09-13T12:06:47.197 INFO:tasks.workunit.client.0.smithi032.stdout: "histogram": [], 2024-09-13T12:06:47.197 INFO:tasks.workunit.client.0.smithi032.stdout: "upper_bound": 1 2024-09-13T12:06:47.197 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.197 INFO:tasks.workunit.client.0.smithi032.stdout: "perf_stat": { 2024-09-13T12:06:47.197 INFO:tasks.workunit.client.0.smithi032.stdout: "commit_latency_ms": 0, 2024-09-13T12:06:47.197 INFO:tasks.workunit.client.0.smithi032.stdout: "apply_latency_ms": 0, 2024-09-13T12:06:47.197 INFO:tasks.workunit.client.0.smithi032.stdout: "commit_latency_ns": 0, 2024-09-13T12:06:47.198 INFO:tasks.workunit.client.0.smithi032.stdout: "apply_latency_ns": 0 2024-09-13T12:06:47.198 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.198 INFO:tasks.workunit.client.0.smithi032.stdout: "alerts": [], 2024-09-13T12:06:47.198 INFO:tasks.workunit.client.0.smithi032.stdout: "network_ping_times": [] 2024-09-13T12:06:47.198 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.198 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_sum_by_class": { 2024-09-13T12:06:47.198 INFO:tasks.workunit.client.0.smithi032.stdout: "hdd": { 2024-09-13T12:06:47.198 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 0, 2024-09-13T12:06:47.198 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 0, 2024-09-13T12:06:47.198 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pgs": 3, 2024-09-13T12:06:47.198 INFO:tasks.workunit.client.0.smithi032.stdout: "num_osds": 3, 2024-09-13T12:06:47.198 INFO:tasks.workunit.client.0.smithi032.stdout: "num_per_pool_osds": 3, 2024-09-13T12:06:47.198 INFO:tasks.workunit.client.0.smithi032.stdout: "num_per_pool_omap_osds": 3, 2024-09-13T12:06:47.198 INFO:tasks.workunit.client.0.smithi032.stdout: "kb": 314572800, 2024-09-13T12:06:47.199 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used": 83644, 2024-09-13T12:06:47.199 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_data": 2104, 2024-09-13T12:06:47.199 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_omap": 10, 2024-09-13T12:06:47.199 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_meta": 81461, 2024-09-13T12:06:47.199 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_avail": 314489156, 2024-09-13T12:06:47.199 INFO:tasks.workunit.client.0.smithi032.stdout: "statfs": { 2024-09-13T12:06:47.199 INFO:tasks.workunit.client.0.smithi032.stdout: "total": 322122547200, 2024-09-13T12:06:47.199 INFO:tasks.workunit.client.0.smithi032.stdout: "available": 322036895744, 2024-09-13T12:06:47.199 INFO:tasks.workunit.client.0.smithi032.stdout: "internally_reserved": 0, 2024-09-13T12:06:47.199 INFO:tasks.workunit.client.0.smithi032.stdout: "allocated": 2154496, 2024-09-13T12:06:47.199 INFO:tasks.workunit.client.0.smithi032.stdout: "data_stored": 417979, 2024-09-13T12:06:47.199 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed": 0, 2024-09-13T12:06:47.199 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_allocated": 0, 2024-09-13T12:06:47.200 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_original": 0, 2024-09-13T12:06:47.200 INFO:tasks.workunit.client.0.smithi032.stdout: "omap_allocated": 10397, 2024-09-13T12:06:47.200 INFO:tasks.workunit.client.0.smithi032.stdout: "internal_metadata": 83416931 2024-09-13T12:06:47.200 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.200 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_peers": [], 2024-09-13T12:06:47.200 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_trim_queue_len": 0, 2024-09-13T12:06:47.200 INFO:tasks.workunit.client.0.smithi032.stdout: "num_snap_trimming": 0, 2024-09-13T12:06:47.200 INFO:tasks.workunit.client.0.smithi032.stdout: "num_shards_repaired": 0, 2024-09-13T12:06:47.200 INFO:tasks.workunit.client.0.smithi032.stdout: "op_queue_age_hist": { 2024-09-13T12:06:47.200 INFO:tasks.workunit.client.0.smithi032.stdout: "histogram": [], 2024-09-13T12:06:47.200 INFO:tasks.workunit.client.0.smithi032.stdout: "upper_bound": 1 2024-09-13T12:06:47.200 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.200 INFO:tasks.workunit.client.0.smithi032.stdout: "perf_stat": { 2024-09-13T12:06:47.200 INFO:tasks.workunit.client.0.smithi032.stdout: "commit_latency_ms": 0, 2024-09-13T12:06:47.201 INFO:tasks.workunit.client.0.smithi032.stdout: "apply_latency_ms": 0, 2024-09-13T12:06:47.201 INFO:tasks.workunit.client.0.smithi032.stdout: "commit_latency_ns": 0, 2024-09-13T12:06:47.201 INFO:tasks.workunit.client.0.smithi032.stdout: "apply_latency_ns": 0 2024-09-13T12:06:47.201 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.201 INFO:tasks.workunit.client.0.smithi032.stdout: "alerts": [], 2024-09-13T12:06:47.201 INFO:tasks.workunit.client.0.smithi032.stdout: "network_ping_times": [] 2024-09-13T12:06:47.201 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.201 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.201 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_stats": [ 2024-09-13T12:06:47.201 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.201 INFO:tasks.workunit.client.0.smithi032.stdout: "poolid": 1, 2024-09-13T12:06:47.201 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg": 1, 2024-09-13T12:06:47.201 INFO:tasks.workunit.client.0.smithi032.stdout: "stat_sum": { 2024-09-13T12:06:47.202 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes": 47971, 2024-09-13T12:06:47.202 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects": 100, 2024-09-13T12:06:47.202 INFO:tasks.workunit.client.0.smithi032.stdout: "num_object_clones": 0, 2024-09-13T12:06:47.202 INFO:tasks.workunit.client.0.smithi032.stdout: "num_object_copies": 300, 2024-09-13T12:06:47.202 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_missing_on_primary": 0, 2024-09-13T12:06:47.202 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_missing": 0, 2024-09-13T12:06:47.202 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_degraded": 100, 2024-09-13T12:06:47.202 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_misplaced": 0, 2024-09-13T12:06:47.202 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_unfound": 0, 2024-09-13T12:06:47.202 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_dirty": 100, 2024-09-13T12:06:47.202 INFO:tasks.workunit.client.0.smithi032.stdout: "num_whiteouts": 0, 2024-09-13T12:06:47.202 INFO:tasks.workunit.client.0.smithi032.stdout: "num_read": 0, 2024-09-13T12:06:47.203 INFO:tasks.workunit.client.0.smithi032.stdout: "num_read_kb": 0, 2024-09-13T12:06:47.203 INFO:tasks.workunit.client.0.smithi032.stdout: "num_write": 101, 2024-09-13T12:06:47.203 INFO:tasks.workunit.client.0.smithi032.stdout: "num_write_kb": 101, 2024-09-13T12:06:47.203 INFO:tasks.workunit.client.0.smithi032.stdout: "num_scrub_errors": 0, 2024-09-13T12:06:47.203 INFO:tasks.workunit.client.0.smithi032.stdout: "num_shallow_scrub_errors": 0, 2024-09-13T12:06:47.203 INFO:tasks.workunit.client.0.smithi032.stdout: "num_deep_scrub_errors": 0, 2024-09-13T12:06:47.203 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_recovered": 0, 2024-09-13T12:06:47.203 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes_recovered": 0, 2024-09-13T12:06:47.203 INFO:tasks.workunit.client.0.smithi032.stdout: "num_keys_recovered": 0, 2024-09-13T12:06:47.203 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_omap": 0, 2024-09-13T12:06:47.203 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_hit_set_archive": 0, 2024-09-13T12:06:47.203 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes_hit_set_archive": 0, 2024-09-13T12:06:47.203 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush": 0, 2024-09-13T12:06:47.203 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_kb": 0, 2024-09-13T12:06:47.203 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict": 0, 2024-09-13T12:06:47.204 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_kb": 0, 2024-09-13T12:06:47.204 INFO:tasks.workunit.client.0.smithi032.stdout: "num_promote": 0, 2024-09-13T12:06:47.204 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_mode_high": 0, 2024-09-13T12:06:47.204 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_mode_low": 0, 2024-09-13T12:06:47.204 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_mode_some": 0, 2024-09-13T12:06:47.204 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_mode_full": 0, 2024-09-13T12:06:47.204 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_pinned": 0, 2024-09-13T12:06:47.204 INFO:tasks.workunit.client.0.smithi032.stdout: "num_legacy_snapsets": 0, 2024-09-13T12:06:47.204 INFO:tasks.workunit.client.0.smithi032.stdout: "num_large_omap_objects": 0, 2024-09-13T12:06:47.204 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_manifest": 0, 2024-09-13T12:06:47.204 INFO:tasks.workunit.client.0.smithi032.stdout: "num_omap_bytes": 0, 2024-09-13T12:06:47.204 INFO:tasks.workunit.client.0.smithi032.stdout: "num_omap_keys": 0, 2024-09-13T12:06:47.204 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_repaired": 0 2024-09-13T12:06:47.205 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.205 INFO:tasks.workunit.client.0.smithi032.stdout: "store_stats": { 2024-09-13T12:06:47.205 INFO:tasks.workunit.client.0.smithi032.stdout: "total": 0, 2024-09-13T12:06:47.205 INFO:tasks.workunit.client.0.smithi032.stdout: "available": 0, 2024-09-13T12:06:47.205 INFO:tasks.workunit.client.0.smithi032.stdout: "internally_reserved": 0, 2024-09-13T12:06:47.205 INFO:tasks.workunit.client.0.smithi032.stdout: "allocated": 1220608, 2024-09-13T12:06:47.205 INFO:tasks.workunit.client.0.smithi032.stdout: "data_stored": 142618, 2024-09-13T12:06:47.205 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed": 0, 2024-09-13T12:06:47.205 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_allocated": 0, 2024-09-13T12:06:47.205 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_original": 0, 2024-09-13T12:06:47.205 INFO:tasks.workunit.client.0.smithi032.stdout: "omap_allocated": 0, 2024-09-13T12:06:47.205 INFO:tasks.workunit.client.0.smithi032.stdout: "internal_metadata": 0 2024-09-13T12:06:47.206 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.206 INFO:tasks.workunit.client.0.smithi032.stdout: "log_size": 17, 2024-09-13T12:06:47.206 INFO:tasks.workunit.client.0.smithi032.stdout: "ondisk_log_size": 17, 2024-09-13T12:06:47.206 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 2, 2024-09-13T12:06:47.206 INFO:tasks.workunit.client.0.smithi032.stdout: "acting": 2, 2024-09-13T12:06:47.206 INFO:tasks.workunit.client.0.smithi032.stdout: "num_store_stats": 3 2024-09-13T12:06:47.206 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.206 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:47.206 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_stats": [ 2024-09-13T12:06:47.206 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.206 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 0, 2024-09-13T12:06:47.207 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 154618822677 2024-09-13T12:06:47.207 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.207 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.207 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 1, 2024-09-13T12:06:47.207 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 42949672991 2024-09-13T12:06:47.207 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.207 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.207 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2, 2024-09-13T12:06:47.207 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 163208757268 2024-09-13T12:06:47.207 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.207 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:47.207 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg_by_state": [ 2024-09-13T12:06:47.207 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.207 INFO:tasks.workunit.client.0.smithi032.stdout: "state": "active+undersized+degraded", 2024-09-13T12:06:47.208 INFO:tasks.workunit.client.0.smithi032.stdout: "num": 1 2024-09-13T12:06:47.208 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.208 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:47.208 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg_by_osd": [ 2024-09-13T12:06:47.208 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.208 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2, 2024-09-13T12:06:47.208 INFO:tasks.workunit.client.0.smithi032.stdout: "num_primary_pg": 0, 2024-09-13T12:06:47.208 INFO:tasks.workunit.client.0.smithi032.stdout: "num_acting_pg": 1, 2024-09-13T12:06:47.208 INFO:tasks.workunit.client.0.smithi032.stdout: "num_up_not_acting_pg": 0 2024-09-13T12:06:47.208 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.208 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.208 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 0, 2024-09-13T12:06:47.208 INFO:tasks.workunit.client.0.smithi032.stdout: "num_primary_pg": 1, 2024-09-13T12:06:47.208 INFO:tasks.workunit.client.0.smithi032.stdout: "num_acting_pg": 1, 2024-09-13T12:06:47.209 INFO:tasks.workunit.client.0.smithi032.stdout: "num_up_not_acting_pg": 0 2024-09-13T12:06:47.209 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.209 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.209 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 1, 2024-09-13T12:06:47.209 INFO:tasks.workunit.client.0.smithi032.stdout: "num_primary_pg": 0, 2024-09-13T12:06:47.209 INFO:tasks.workunit.client.0.smithi032.stdout: "num_acting_pg": 0, 2024-09-13T12:06:47.209 INFO:tasks.workunit.client.0.smithi032.stdout: "num_up_not_acting_pg": 0 2024-09-13T12:06:47.209 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.209 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:47.209 INFO:tasks.workunit.client.0.smithi032.stdout: "purged_snaps": [ 2024-09-13T12:06:47.209 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:06:47.209 INFO:tasks.workunit.client.0.smithi032.stdout: "pool": 1, 2024-09-13T12:06:47.209 INFO:tasks.workunit.client.0.smithi032.stdout: "purged_snaps": {} 2024-09-13T12:06:47.210 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.210 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:06:47.210 INFO:tasks.workunit.client.0.smithi032.stdout: "servicemap": { 2024-09-13T12:06:47.210 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 6, 2024-09-13T12:06:47.210 INFO:tasks.workunit.client.0.smithi032.stdout: "modified": "2024-09-13T12:06:29.139614+0000", 2024-09-13T12:06:47.210 INFO:tasks.workunit.client.0.smithi032.stdout: "services": {} 2024-09-13T12:06:47.210 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:06:47.210 INFO:tasks.workunit.client.0.smithi032.stdout: "mgrstat_first_committed": 1, 2024-09-13T12:06:47.210 INFO:tasks.workunit.client.0.smithi032.stdout: "mgrstat_last_committed": 111, 2024-09-13T12:06:47.210 INFO:tasks.workunit.client.0.smithi032.stdout: "logm_first_committed": 1, 2024-09-13T12:06:47.210 INFO:tasks.workunit.client.0.smithi032.stdout: "logm_last_committed": 449, 2024-09-13T12:06:47.210 INFO:tasks.workunit.client.0.smithi032.stdout: "paxos": { 2024-09-13T12:06:47.210 INFO:tasks.workunit.client.0.smithi032.stdout: "first_committed": 1, 2024-09-13T12:06:47.210 INFO:tasks.workunit.client.0.smithi032.stdout: "last_committed": 607, 2024-09-13T12:06:47.211 INFO:tasks.workunit.client.0.smithi032.stdout: "last_pn": 0, 2024-09-13T12:06:47.211 INFO:tasks.workunit.client.0.smithi032.stdout: "accepted_pn": 0 2024-09-13T12:06:47.211 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:06:47.211 INFO:tasks.workunit.client.0.smithi032.stdout:} 2024-09-13T12:06:47.211 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-13T12:06:47.211 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2024-09-13T12:06:47.211 INFO:tasks.workunit.client.0.smithi032.stdout:delay recovery 2024-09-13T12:06:47.212 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2024-09-13T12:06:47.539 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:06:47.539 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-13T12:05:31.913985+0000 41'101 41:131 [0,2] 0 [0,2] 0 0'0 2024-09-13T12:04:30.066569+0000 0'0 2024-09-13T12:04:30.066569+0000 0 0 periodic scrub scheduled @ 2024-09-14T13:20:53.243442+0000 0 0 2024-09-13T12:06:47.539 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:06:47.539 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:06:47.540 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:06:47.550 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-09-13T12:06:47.551 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2024-09-13T12:06:47.551 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-13T12:06:47.551 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-13T12:06:47.551 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:06:47.551 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:06:47.551 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:06:47.551 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-osd.0.asok 2024-09-13T12:06:47.552 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-09-13T12:06:47.552 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.33242/ceph-osd.0.asok set_recovery_delay 100000 2024-09-13T12:06:47.635 INFO:tasks.workunit.client.0.smithi032.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-13T12:06:47.643 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-09-13T12:06:47.643 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2024-09-13T12:06:47.643 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-13T12:06:47.643 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-13T12:06:47.644 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:06:47.644 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:06:47.644 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:06:47.644 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-osd.2.asok 2024-09-13T12:06:47.644 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-09-13T12:06:47.644 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.33242/ceph-osd.2.asok set_recovery_delay 100000 2024-09-13T12:06:47.728 INFO:tasks.workunit.client.0.smithi032.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-13T12:06:47.735 INFO:tasks.workunit.client.0.smithi032.stdout:revive divergent 1 2024-09-13T12:06:47.735 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2024-09-13T12:06:47.735 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2024-09-13T12:06:48.124 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:06:48.124 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-13T12:05:31.913985+0000 41'101 41:131 [0,2] 0 [0,2] 0 0'0 2024-09-13T12:04:30.066569+0000 0'0 2024-09-13T12:04:30.066569+0000 0 0 periodic scrub scheduled @ 2024-09-14T13:20:53.243442+0000 0 0 2024-09-13T12:06:48.124 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:06:48.124 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:06:48.124 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:06:48.135 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2024-09-13T12:06:48.777 INFO:tasks.workunit.client.0.smithi032.stderr:noup is set 2024-09-13T12:06:48.792 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-09-13T12:06:48.792 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-13T12:06:48.792 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:06:48.792 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T12:06:48.792 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:06:48.792 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-13T12:06:48.792 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T12:06:48.793 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:06:48.793 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:06:48.793 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:06:48.793 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-13T12:06:48.793 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-13T12:06:48.793 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:06:48.793 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:06:48.793 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T12:06:48.793 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:06:48.793 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:06:48.793 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:06:48.793 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:06:48.794 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:06:48.794 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:06:48.794 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:06:48.794 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:06:48.794 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:06:48.794 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T12:06:48.794 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T12:06:48.794 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:06:48.794 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:06:48.794 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:06:48.794 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:06:48.794 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:06:48.794 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:06:48.795 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-13T12:06:48.795 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.1 2024-09-13T12:06:48.796 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T12:06:48.796 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:06:48.796 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-13T12:06:48.796 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T12:06:48.796 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:06:48.796 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:06:48.796 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:06:48.829 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:06:48.828+0000 7fbad6132640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:06:48.837 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:06:48.836+0000 7fbad6132640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:06:48.838 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:06:48.837+0000 7fbad6132640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:06:49.250 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: return 0 2024-09-13T12:06:49.250 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2024-09-13T12:06:49.695 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:06:49.694+0000 7fbad6132640 -1 Falling back to public interface 2024-09-13T12:06:53.184 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:06:53.183+0000 7fbad6132640 -1 osd.1 29 log_to_monitors true 2024-09-13T12:06:54.251 INFO:tasks.workunit.client.0.smithi032.stdout:delay recovery divergent 2024-09-13T12:06:54.251 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2024-09-13T12:06:54.251 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2024-09-13T12:06:54.651 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:06:54.651 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-13T12:05:31.913985+0000 41'101 43:135 [0,2] 0 [0,2] 0 0'0 2024-09-13T12:04:30.066569+0000 0'0 2024-09-13T12:04:30.066569+0000 0 0 periodic scrub scheduled @ 2024-09-14T13:20:53.243442+0000 0 0 2024-09-13T12:06:54.651 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:06:54.651 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:06:54.651 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:06:54.663 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2024-09-13T12:06:54.663 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-13T12:06:54.663 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-13T12:06:54.664 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:06:54.664 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:06:54.664 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:06:54.664 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-osd.1.asok 2024-09-13T12:06:54.664 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2024-09-13T12:06:54.664 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.33242/ceph-osd.1.asok set_recovery_delay 100000 2024-09-13T12:06:54.747 INFO:tasks.workunit.client.0.smithi032.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-13T12:06:54.754 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2024-09-13T12:06:55.483 INFO:tasks.workunit.client.0.smithi032.stderr:noup is unset 2024-09-13T12:06:55.495 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2024-09-13T12:06:55.495 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:06:55.495 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T12:06:55.495 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:06:55.495 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:06:55.495 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:06:55.495 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:06:55.496 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:06:55.496 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:06:55.496 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:06:55.948 INFO:tasks.workunit.client.0.smithi032.stdout:osd.0 up in weight 1 up_from 36 up_thru 40 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/3817518293,v1:127.0.0.1:6803/3817518293] [v2:127.0.0.1:6804/3817518293,v1:127.0.0.1:6805/3817518293] exists,up 940cd0ac-d018-458e-a8bb-6eb19147770c 2024-09-13T12:06:55.949 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:06:55.949 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:06:55.950 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:06:55.950 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2024-09-13T12:06:55.950 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:06:55.950 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T12:06:55.950 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:06:55.950 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:06:55.950 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:06:55.951 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:06:55.951 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:06:55.952 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:06:55.952 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:06:56.389 INFO:tasks.workunit.client.0.smithi032.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/4036056442,v1:127.0.0.1:6819/4036056442] [v2:127.0.0.1:6820/4036056442,v1:127.0.0.1:6821/4036056442] exists,up eb913216-75c2-4cc2-ad3b-855fbbad0fc3 2024-09-13T12:06:56.390 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:06:56.390 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:06:56.390 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:06:56.390 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2024-09-13T12:06:56.390 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:06:56.390 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T12:06:56.390 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:06:56.390 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:06:56.391 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:06:56.391 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:06:56.391 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:06:56.391 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:06:56.391 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:06:56.861 INFO:tasks.workunit.client.0.smithi032.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/746434650,v1:127.0.0.1:6811/746434650] [v2:127.0.0.1:6812/746434650,v1:127.0.0.1:6813/746434650] exists,up 69901422-110f-4bc1-8cb2-72564b47a69b 2024-09-13T12:06:56.862 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:06:56.862 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:06:56.862 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:06:56.862 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2024-09-13T12:06:57.254 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:06:57.254 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-13T12:05:31.913985+0000 41'101 44:137 [0,2] 0 [0,2] 0 0'0 2024-09-13T12:04:30.066569+0000 0'0 2024-09-13T12:04:30.066569+0000 0 0 periodic scrub scheduled @ 2024-09-14T13:20:53.243442+0000 0 0 2024-09-13T12:06:57.254 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:06:57.254 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:06:57.254 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:06:57.265 INFO:tasks.workunit.client.0.smithi032.stdout:wait for peering 2024-09-13T12:06:57.265 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2024-09-13T12:06:57.265 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2024-09-13T12:06:57.652 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:06:57.652 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-13T12:05:31.913985+0000 41'101 44:137 [0,2] 0 [0,2] 0 0'0 2024-09-13T12:04:30.066569+0000 0'0 2024-09-13T12:04:30.066569+0000 0 0 periodic scrub scheduled @ 2024-09-14T13:20:53.243442+0000 0 0 2024-09-13T12:06:57.653 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:06:57.653 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:06:57.653 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:06:57.663 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2024-09-13T12:06:57.810 INFO:tasks.workunit.client.0.smithi032.stdout:killing divergent 1 2024-09-13T12:06:57.811 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2024-09-13T12:06:57.811 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2024-09-13T12:06:58.203 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:06:58.204 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-13T12:05:31.913985+0000 41'101 44:137 [0,2] 0 [0,2] 0 0'0 2024-09-13T12:04:30.066569+0000 0'0 2024-09-13T12:04:30.066569+0000 0 0 periodic scrub scheduled @ 2024-09-14T13:20:53.243442+0000 0 0 2024-09-13T12:06:58.204 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:06:58.204 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:06:58.204 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:06:58.215 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2024-09-13T12:06:58.215 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:06:58.215 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:06:58.215 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:06:58.216 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:06:58.216 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:06:58.321 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:06:58.321 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.33242.out 2024-09-13T12:06:58.322 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.33242.out 2024-09-13T12:06:58.322 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-13T12:06:58.322 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T12:06:58.322 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T12:06:58.322 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T12:06:58.322 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-13T12:06:58.322 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.33242.out 2024-09-13T12:06:59.814 INFO:tasks.workunit.client.0.smithi032.stderr:Exporting 1.0 info 1.0( v 47'102 lc 0'0 (29'90,47'102] local-lis/les=46/47 n=101 ec=23/23 lis/c=46/23 les/c/f=47/24/0 sis=46) 2024-09-13T12:06:59.815 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:023dd3d2:::existing_12:head# 2024-09-13T12:06:59.815 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:04c40a0c:::existing_93:head# 2024-09-13T12:06:59.817 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:064ff5fb:::existing_36:head# 2024-09-13T12:06:59.817 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:091f9f1d:::existing_94:head# 2024-09-13T12:06:59.817 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:0a865a93:::existing_54:head# 2024-09-13T12:06:59.817 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:0fed513c:::existing_41:head# 2024-09-13T12:06:59.817 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:13b9e0b8:::existing_92:head# 2024-09-13T12:06:59.819 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:169157fa:::existing_81:head# 2024-09-13T12:06:59.819 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:17731382:::existing_58:head# 2024-09-13T12:06:59.819 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:18a45a39:::existing_43:head# 2024-09-13T12:06:59.819 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:1c160f01:::existing_66:head# 2024-09-13T12:06:59.819 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:1dd6c249:::existing_97:head# 2024-09-13T12:06:59.820 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:227e0c77:::existing_7:head# 2024-09-13T12:06:59.820 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:23e1b063:::existing_59:head# 2024-09-13T12:06:59.820 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:28566edb:::existing_78:head# 2024-09-13T12:06:59.820 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:2a101182:::existing_20:head# 2024-09-13T12:06:59.820 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:2d1519da:::existing_96:head# 2024-09-13T12:06:59.822 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:3078b87a:::existing_86:head# 2024-09-13T12:06:59.822 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:30bde01b:::existing_15:head# 2024-09-13T12:06:59.822 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:317a7e19:::existing_56:head# 2024-09-13T12:06:59.822 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:38b9c7fa:::existing_33:head# 2024-09-13T12:06:59.822 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:3e16d84b:::existing_48:head# 2024-09-13T12:06:59.822 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:45fe8203:::existing_75:head# 2024-09-13T12:06:59.822 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:4e82c9ba:::existing_50:head# 2024-09-13T12:06:59.823 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:5298dcdb:::existing_57:head# 2024-09-13T12:06:59.826 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:52ebaa44:::existing_32:head# 2024-09-13T12:06:59.826 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:53588912:::existing_9:head# 2024-09-13T12:06:59.826 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:5b7e2de6:::existing_84:head# 2024-09-13T12:06:59.826 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:5da0255c:::existing_13:head# 2024-09-13T12:06:59.826 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:5da2dab8:::existing_53:head# 2024-09-13T12:06:59.826 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:5ee3514b:::existing_30:head# 2024-09-13T12:06:59.826 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:602f83fe:::foo:head# 2024-09-13T12:06:59.827 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:60426cc7:::existing_62:head# 2024-09-13T12:06:59.827 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:628918ca:::existing_79:head# 2024-09-13T12:06:59.827 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:63f8ab2d:::existing_77:head# 2024-09-13T12:06:59.827 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:67e1780f:::existing_27:head# 2024-09-13T12:06:59.827 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:68143ca1:::existing_71:head# 2024-09-13T12:06:59.827 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:6bac9509:::existing_29:head# 2024-09-13T12:06:59.827 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:6bbec798:::existing_45:head# 2024-09-13T12:06:59.827 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:6cb23a40:::existing_65:head# 2024-09-13T12:06:59.827 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:7055ec5e:::existing_83:head# 2024-09-13T12:06:59.827 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:70a54e2b:::existing_87:head# 2024-09-13T12:06:59.827 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:750ace21:::existing_19:head# 2024-09-13T12:06:59.827 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:78623b1d:::existing_6:head# 2024-09-13T12:06:59.833 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:79881152:::existing_63:head# 2024-09-13T12:06:59.833 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:7a80c0d5:::existing_99:head# 2024-09-13T12:06:59.833 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:81490677:::existing_76:head# 2024-09-13T12:06:59.833 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:85ec8e5d:::existing_85:head# 2024-09-13T12:06:59.833 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:8e2abbf0:::existing_49:head# 2024-09-13T12:06:59.833 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:8ee0fe0f:::existing_52:head# 2024-09-13T12:06:59.833 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:923c5d4d:::existing_5:head# 2024-09-13T12:06:59.833 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:92731e71:::existing_89:head# 2024-09-13T12:06:59.834 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:928205cf:::existing_82:head# 2024-09-13T12:06:59.834 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:93be2267:::existing_24:head# 2024-09-13T12:06:59.834 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:99ceda2c:::existing_44:head# 2024-09-13T12:06:59.834 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:9dd917ee:::existing_26:head# 2024-09-13T12:06:59.834 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:a28805a6:::existing_72:head# 2024-09-13T12:06:59.834 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:a294a8f6:::existing_16:head# 2024-09-13T12:06:59.834 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:a2f5473a:::existing_42:head# 2024-09-13T12:06:59.834 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:a504a533:::existing_90:head# 2024-09-13T12:06:59.834 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:a691337e:::existing_61:head# 2024-09-13T12:06:59.834 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:a6e8d7ac:::existing_28:head# 2024-09-13T12:06:59.834 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:a9fde86e:::existing_18:head# 2024-09-13T12:06:59.834 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:aa309faa:::existing_68:head# 2024-09-13T12:06:59.834 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:ae40b0ed:::existing_31:head# 2024-09-13T12:06:59.834 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:b279b1fe:::existing_10:head# 2024-09-13T12:06:59.835 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:b343e277:::existing_88:head# 2024-09-13T12:06:59.835 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:b7130e90:::existing_80:head# 2024-09-13T12:06:59.835 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:bc6f7309:::existing_70:head# 2024-09-13T12:06:59.835 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:be8489b7:::existing_38:head# 2024-09-13T12:06:59.835 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:beb4e5e3:::existing_47:head# 2024-09-13T12:06:59.835 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:bebec877:::existing_74:head# 2024-09-13T12:06:59.835 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:c04347d8:::existing_60:head# 2024-09-13T12:06:59.835 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:c156033e:::existing_100:head# 2024-09-13T12:06:59.839 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:c19f5a8e:::existing_8:head# 2024-09-13T12:06:59.839 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:c3e6ebe4:::existing_17:head# 2024-09-13T12:06:59.839 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:c6c5a7c2:::existing_39:head# 2024-09-13T12:06:59.839 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:cc1d0abf:::existing_95:head# 2024-09-13T12:06:59.839 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:ccb3865b:::existing_73:head# 2024-09-13T12:06:59.840 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:cd84e19e:::existing_40:head# 2024-09-13T12:06:59.840 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:d2cd64ce:::existing_37:head# 2024-09-13T12:06:59.840 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:d41a25c9:::existing_22:head# 2024-09-13T12:06:59.840 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:d7f088da:::existing_91:head# 2024-09-13T12:06:59.840 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-09-13T12:06:59.840 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-09-13T12:06:59.840 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:dd677bd3:::existing_35:head# 2024-09-13T12:06:59.840 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-09-13T12:06:59.840 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:e0e22077:::existing_21:head# 2024-09-13T12:06:59.840 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:e255b710:::existing_55:head# 2024-09-13T12:06:59.840 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:eff1d5db:::existing_14:head# 2024-09-13T12:06:59.840 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:f1b156bf:::existing_23:head# 2024-09-13T12:06:59.840 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:f5ce4199:::existing_34:head# 2024-09-13T12:06:59.840 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:f6809456:::existing_11:head# 2024-09-13T12:06:59.841 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:f699a2a9:::existing_67:head# 2024-09-13T12:06:59.841 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:f9baac43:::existing_25:head# 2024-09-13T12:06:59.841 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:fb76688e:::existing_51:head# 2024-09-13T12:06:59.841 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:fbb92eec:::existing_69:head# 2024-09-13T12:06:59.841 INFO:tasks.workunit.client.0.smithi032.stderr:Export successful 2024-09-13T12:06:59.841 INFO:tasks.workunit.client.0.smithi032.stdout: marking collection for removal 2024-09-13T12:06:59.841 INFO:tasks.workunit.client.0.smithi032.stdout:setting '_remove' omap key 2024-09-13T12:06:59.842 INFO:tasks.workunit.client.0.smithi032.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-09-13T12:07:00.073 INFO:tasks.workunit.client.0.smithi032.stderr:Remove successful 2024-09-13T12:07:00.354 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.33242.out 2024-09-13T12:07:00.355 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-13T12:07:00.355 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T12:07:00.355 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T12:07:00.355 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T12:07:00.355 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-13T12:07:00.355 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.33242.out 2024-09-13T12:07:01.503 INFO:tasks.workunit.client.0.smithi032.stdout:get_pg_num_history pg_num_history pg_num_history(e47 pg_nums {1={23=1}} deleted_pools ) 2024-09-13T12:07:01.503 INFO:tasks.workunit.client.0.smithi032.stdout:Importing pgid 1.0 2024-09-13T12:07:01.503 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:023dd3d2:::existing_12:head# 2024-09-13T12:07:01.503 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:01.568 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:04c40a0c:::existing_93:head# 2024-09-13T12:07:01.568 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:01.601 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:064ff5fb:::existing_36:head# 2024-09-13T12:07:01.601 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:01.635 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:091f9f1d:::existing_94:head# 2024-09-13T12:07:01.635 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:01.668 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:0a865a93:::existing_54:head# 2024-09-13T12:07:01.668 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:01.702 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:0fed513c:::existing_41:head# 2024-09-13T12:07:01.702 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:01.735 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:13b9e0b8:::existing_92:head# 2024-09-13T12:07:01.735 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:01.769 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:169157fa:::existing_81:head# 2024-09-13T12:07:01.769 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:01.802 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:17731382:::existing_58:head# 2024-09-13T12:07:01.802 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:01.827 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:18a45a39:::existing_43:head# 2024-09-13T12:07:01.827 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:01.852 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:1c160f01:::existing_66:head# 2024-09-13T12:07:01.853 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:01.878 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:1dd6c249:::existing_97:head# 2024-09-13T12:07:01.878 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:01.903 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:227e0c77:::existing_7:head# 2024-09-13T12:07:01.903 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:01.928 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:23e1b063:::existing_59:head# 2024-09-13T12:07:01.928 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:01.953 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:28566edb:::existing_78:head# 2024-09-13T12:07:01.953 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:01.978 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:2a101182:::existing_20:head# 2024-09-13T12:07:01.978 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.003 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:2d1519da:::existing_96:head# 2024-09-13T12:07:02.003 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.028 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:3078b87a:::existing_86:head# 2024-09-13T12:07:02.028 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.053 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:30bde01b:::existing_15:head# 2024-09-13T12:07:02.053 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.078 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:317a7e19:::existing_56:head# 2024-09-13T12:07:02.078 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.145 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:38b9c7fa:::existing_33:head# 2024-09-13T12:07:02.146 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.213 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:3e16d84b:::existing_48:head# 2024-09-13T12:07:02.213 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.263 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:45fe8203:::existing_75:head# 2024-09-13T12:07:02.263 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.288 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:4e82c9ba:::existing_50:head# 2024-09-13T12:07:02.288 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.313 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:5298dcdb:::existing_57:head# 2024-09-13T12:07:02.313 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.338 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:52ebaa44:::existing_32:head# 2024-09-13T12:07:02.338 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.363 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:53588912:::existing_9:head# 2024-09-13T12:07:02.363 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.388 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:5b7e2de6:::existing_84:head# 2024-09-13T12:07:02.388 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.413 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:5da0255c:::existing_13:head# 2024-09-13T12:07:02.413 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.439 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:5da2dab8:::existing_53:head# 2024-09-13T12:07:02.439 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.464 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:5ee3514b:::existing_30:head# 2024-09-13T12:07:02.464 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.506 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:602f83fe:::foo:head# 2024-09-13T12:07:02.506 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.539 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:60426cc7:::existing_62:head# 2024-09-13T12:07:02.539 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.573 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:628918ca:::existing_79:head# 2024-09-13T12:07:02.573 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.606 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:63f8ab2d:::existing_77:head# 2024-09-13T12:07:02.606 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.640 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:67e1780f:::existing_27:head# 2024-09-13T12:07:02.640 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.673 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:68143ca1:::existing_71:head# 2024-09-13T12:07:02.673 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.707 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:6bac9509:::existing_29:head# 2024-09-13T12:07:02.707 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.757 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:6bbec798:::existing_45:head# 2024-09-13T12:07:02.757 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.790 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:6cb23a40:::existing_65:head# 2024-09-13T12:07:02.790 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.824 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:7055ec5e:::existing_83:head# 2024-09-13T12:07:02.824 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.857 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:70a54e2b:::existing_87:head# 2024-09-13T12:07:02.858 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.891 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:750ace21:::existing_19:head# 2024-09-13T12:07:02.891 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.924 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:78623b1d:::existing_6:head# 2024-09-13T12:07:02.924 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.957 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:79881152:::existing_63:head# 2024-09-13T12:07:02.958 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:02.991 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:7a80c0d5:::existing_99:head# 2024-09-13T12:07:02.991 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:03.033 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:81490677:::existing_76:head# 2024-09-13T12:07:03.033 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:03.067 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:85ec8e5d:::existing_85:head# 2024-09-13T12:07:03.067 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:03.100 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:8e2abbf0:::existing_49:head# 2024-09-13T12:07:03.100 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:03.133 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:8ee0fe0f:::existing_52:head# 2024-09-13T12:07:03.133 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:03.167 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:923c5d4d:::existing_5:head# 2024-09-13T12:07:03.167 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:03.200 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:92731e71:::existing_89:head# 2024-09-13T12:07:03.200 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:03.267 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:928205cf:::existing_82:head# 2024-09-13T12:07:03.267 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:03.317 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:93be2267:::existing_24:head# 2024-09-13T12:07:03.318 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:03.351 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:99ceda2c:::existing_44:head# 2024-09-13T12:07:03.351 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:03.384 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:9dd917ee:::existing_26:head# 2024-09-13T12:07:03.385 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:03.418 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:a28805a6:::existing_72:head# 2024-09-13T12:07:03.418 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:03.451 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:a294a8f6:::existing_16:head# 2024-09-13T12:07:03.451 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:03.485 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:a2f5473a:::existing_42:head# 2024-09-13T12:07:03.485 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:03.518 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:a504a533:::existing_90:head# 2024-09-13T12:07:03.518 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:03.552 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:a691337e:::existing_61:head# 2024-09-13T12:07:03.552 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:03.585 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:a6e8d7ac:::existing_28:head# 2024-09-13T12:07:03.585 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:03.619 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:a9fde86e:::existing_18:head# 2024-09-13T12:07:03.619 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:03.652 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:aa309faa:::existing_68:head# 2024-09-13T12:07:03.652 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:03.686 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:ae40b0ed:::existing_31:head# 2024-09-13T12:07:03.687 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:03.752 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:b279b1fe:::existing_10:head# 2024-09-13T12:07:03.753 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:03.786 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:b343e277:::existing_88:head# 2024-09-13T12:07:03.786 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:03.828 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:b7130e90:::existing_80:head# 2024-09-13T12:07:03.828 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:03.861 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:bc6f7309:::existing_70:head# 2024-09-13T12:07:03.861 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:03.894 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:be8489b7:::existing_38:head# 2024-09-13T12:07:03.895 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:03.928 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:beb4e5e3:::existing_47:head# 2024-09-13T12:07:03.928 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:03.986 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:bebec877:::existing_74:head# 2024-09-13T12:07:03.987 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:04.020 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:c04347d8:::existing_60:head# 2024-09-13T12:07:04.020 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:04.053 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:c156033e:::existing_100:head# 2024-09-13T12:07:04.053 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:04.079 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:c19f5a8e:::existing_8:head# 2024-09-13T12:07:04.079 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:04.145 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:c3e6ebe4:::existing_17:head# 2024-09-13T12:07:04.145 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:04.196 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:c6c5a7c2:::existing_39:head# 2024-09-13T12:07:04.196 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:04.246 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:cc1d0abf:::existing_95:head# 2024-09-13T12:07:04.246 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:04.271 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:ccb3865b:::existing_73:head# 2024-09-13T12:07:04.271 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:04.296 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:cd84e19e:::existing_40:head# 2024-09-13T12:07:04.296 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:04.321 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:d2cd64ce:::existing_37:head# 2024-09-13T12:07:04.322 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:04.346 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:d41a25c9:::existing_22:head# 2024-09-13T12:07:04.346 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:04.372 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:d7f088da:::existing_91:head# 2024-09-13T12:07:04.372 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:04.397 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-09-13T12:07:04.397 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:04.422 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-09-13T12:07:04.422 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:04.447 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:dd677bd3:::existing_35:head# 2024-09-13T12:07:04.447 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:04.481 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-09-13T12:07:04.481 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:04.506 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:e0e22077:::existing_21:head# 2024-09-13T12:07:04.506 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:04.531 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:e255b710:::existing_55:head# 2024-09-13T12:07:04.531 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:04.556 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:eff1d5db:::existing_14:head# 2024-09-13T12:07:04.556 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:04.581 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:f1b156bf:::existing_23:head# 2024-09-13T12:07:04.582 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:04.615 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:f5ce4199:::existing_34:head# 2024-09-13T12:07:04.615 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:04.640 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:f6809456:::existing_11:head# 2024-09-13T12:07:04.640 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:04.673 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:f699a2a9:::existing_67:head# 2024-09-13T12:07:04.674 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:04.707 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:f9baac43:::existing_25:head# 2024-09-13T12:07:04.707 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:04.740 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:fb76688e:::existing_51:head# 2024-09-13T12:07:04.740 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:04.774 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:fbb92eec:::existing_69:head# 2024-09-13T12:07:04.774 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:07:04.807 INFO:tasks.workunit.client.0.smithi032.stdout:write_pg epoch 47 info 1.0( v 47'102 lc 0'0 (29'90,47'102] local-lis/les=46/47 n=101 ec=23/23 lis/c=46/23 les/c/f=47/24/0 sis=46) 2024-09-13T12:07:04.841 INFO:tasks.workunit.client.0.smithi032.stdout:Import successful 2024-09-13T12:07:05.357 INFO:tasks.workunit.client.0.smithi032.stdout:reviving divergent 1 2024-09-13T12:07:05.358 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2024-09-13T12:07:05.358 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2024-09-13T12:07:05.757 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:07:05.757 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-13T12:05:31.913985+0000 41'101 44:137 [0,2] 0 [0,2] 0 0'0 2024-09-13T12:04:30.066569+0000 0'0 2024-09-13T12:04:30.066569+0000 0 0 periodic scrub scheduled @ 2024-09-14T13:20:53.243442+0000 0 0 2024-09-13T12:07:05.757 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:07:05.757 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:07:05.757 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:07:05.770 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-09-13T12:07:05.770 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-13T12:07:05.770 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:07:05.770 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T12:07:05.770 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:07:05.770 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-13T12:07:05.771 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T12:07:05.771 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:07:05.771 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:07:05.771 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:07:05.771 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-13T12:07:05.771 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-13T12:07:05.771 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:07:05.771 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:07:05.771 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T12:07:05.771 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:07:05.771 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:07:05.771 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:07:05.771 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:07:05.771 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:07:05.772 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:07:05.772 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:07:05.772 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:07:05.772 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:07:05.772 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T12:07:05.772 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T12:07:05.772 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:07:05.772 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:07:05.773 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:07:05.773 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:07:05.773 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:07:05.773 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:07:05.773 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-13T12:07:05.773 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.1 2024-09-13T12:07:05.774 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T12:07:05.774 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:07:05.774 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-13T12:07:05.774 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T12:07:05.775 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:07:05.775 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:07:05.775 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:07:05.807 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:07:05.805+0000 7f265bc2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:07:05.807 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:07:05.807+0000 7f265bc2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:07:05.808 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:07:05.808+0000 7f265bc2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:07:06.180 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T12:07:06.180 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:07:06.180 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T12:07:06.180 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:07:06.181 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:07:06.181 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:07:06.181 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:07:06.181 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:07:06.181 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:07:06.182 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:07:06.625 INFO:tasks.workunit.client.0.smithi032.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/4036056442,v1:127.0.0.1:6819/4036056442] [v2:127.0.0.1:6820/4036056442,v1:127.0.0.1:6821/4036056442] exists,up eb913216-75c2-4cc2-ad3b-855fbbad0fc3 2024-09-13T12:07:06.626 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:07:06.626 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:07:06.626 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:07:06.626 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2024-09-13T12:07:06.626 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:07:06.626 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T12:07:06.626 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:07:06.626 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:07:06.626 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:07:06.626 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:07:06.627 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:07:06.627 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:07:06.627 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:07:07.065 INFO:tasks.workunit.client.0.smithi032.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/4036056442,v1:127.0.0.1:6819/4036056442] [v2:127.0.0.1:6820/4036056442,v1:127.0.0.1:6821/4036056442] exists,up eb913216-75c2-4cc2-ad3b-855fbbad0fc3 2024-09-13T12:07:07.065 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:07:07.066 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:07:07.066 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:07:07.066 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2024-09-13T12:07:07.168 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:07:07.167+0000 7f265bc2c640 -1 Falling back to public interface 2024-09-13T12:07:10.192 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:07:10.191+0000 7f265bc2c640 -1 osd.1 47 log_to_monitors true 2024-09-13T12:07:27.068 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2024-09-13T12:07:27.100 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-13T12:07:27.100 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-13T12:07:27.100 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:07:27.100 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:07:27.100 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:07:27.100 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-osd.1.asok 2024-09-13T12:07:27.100 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2024-09-13T12:07:27.100 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.33242/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T12:07:27.160 INFO:tasks.workunit.client.0.smithi032.stdout:{ 2024-09-13T12:07:27.160 INFO:tasks.workunit.client.0.smithi032.stdout: "ops": [], 2024-09-13T12:07:27.160 INFO:tasks.workunit.client.0.smithi032.stdout: "num_ops": 0 2024-09-13T12:07:27.160 INFO:tasks.workunit.client.0.smithi032.stdout:} 2024-09-13T12:07:27.168 INFO:tasks.workunit.client.0.smithi032.stdout:allowing recovery 2024-09-13T12:07:27.168 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2024-09-13T12:07:27.168 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2024-09-13T12:07:27.557 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:07:27.557 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 101 0 0 0 0 48454 0 0 12 90 12 active+clean 2024-09-13T12:07:12.044190+0000 47'102 51:157 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-13T12:04:30.066569+0000 0'0 2024-09-13T12:04:30.066569+0000 0 0 periodic scrub scheduled @ 2024-09-14T21:23:57.454459+0000 0 0 2024-09-13T12:07:27.557 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:07:27.558 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:07:27.558 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:07:27.568 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-09-13T12:07:27.569 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-13T12:07:27.684 INFO:tasks.workunit.client.0.smithi032.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-13T12:07:27.693 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-09-13T12:07:27.693 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-13T12:07:27.808 INFO:tasks.workunit.client.0.smithi032.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-13T12:07:27.817 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-09-13T12:07:27.817 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-13T12:07:27.934 INFO:tasks.workunit.client.0.smithi032.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-13T12:07:27.943 INFO:tasks.workunit.client.0.smithi032.stdout:reading divergent objects 2024-09-13T12:07:27.943 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2024-09-13T12:07:27.943 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2024-09-13T12:07:28.333 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:07:28.333 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 101 0 0 0 0 48454 0 0 12 90 12 active+clean 2024-09-13T12:07:12.044190+0000 47'102 51:157 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-13T12:04:30.066569+0000 0'0 2024-09-13T12:04:30.066569+0000 0 0 periodic scrub scheduled @ 2024-09-14T21:23:57.454459+0000 0 0 2024-09-13T12:07:28.333 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:07:28.334 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:07:28.334 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:07:28.345 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2024-09-13T12:07:28.345 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2024-09-13T12:07:28.346 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-13T12:07:28.346 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2024-09-13T12:07:28.383 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-13T12:07:28.383 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2024-09-13T12:07:28.420 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-13T12:07:28.420 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2024-09-13T12:07:28.457 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-13T12:07:28.457 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2024-09-13T12:07:28.493 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2024-09-13T12:07:28.494 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-13T12:07:28.494 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2024-09-13T12:07:28.528 INFO:tasks.workunit.client.0.smithi032.stderr:error getting test/newobject_1: (2) No such file or directory 2024-09-13T12:07:28.530 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-13T12:07:28.530 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2024-09-13T12:07:28.565 INFO:tasks.workunit.client.0.smithi032.stderr:error getting test/newobject_2: (2) No such file or directory 2024-09-13T12:07:28.567 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2024-09-13T12:07:28.568 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-09-13T12:07:28.569 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-13T12:07:28.574 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 29'105 (29'4) delete 1:70b3bf12:::existing_4:head by client.4638.0:1 2024-09-13T12:05:05.731841+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-13T12:07:28.574 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 20 _merge_object_divergent_entries: keeping 29'105 (29'4) delete 1:70b3bf12:::existing_4:head by client.4638.0:1 2024-09-13T12:05:05.731841+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-13T12:07:28.574 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-13T12:07:28.574 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 29'4 first_divergent_update: 29'105 last_divergent_update: 29'105 2024-09-13T12:07:28.574 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: more recent entry found: 41'101 (29'4) modify 1:70b3bf12:::existing_4:head by client.4756.0:1 2024-09-13T12:06:09.394826+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false, already merged 2024-09-13T12:07:28.574 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 29'103 (0'0) modify 1:75ac5091:::newobject_2:head by client.4631.0:1 2024-09-13T12:05:05.725929+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-13T12:07:28.574 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 20 _merge_object_divergent_entries: keeping 29'103 (0'0) modify 1:75ac5091:::newobject_2:head by client.4631.0:1 2024-09-13T12:05:05.725929+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-13T12:07:28.574 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-13T12:07:28.574 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 29'103 last_divergent_update: 29'103 2024-09-13T12:07:28.575 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-09-13T12:07:28.575 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-09-13T12:07:28.575 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 29'106 (29'3) delete 1:8881e6d8:::existing_3:head by client.4639.0:1 2024-09-13T12:05:05.733761+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-13T12:07:28.575 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 20 _merge_object_divergent_entries: keeping 29'106 (29'3) delete 1:8881e6d8:::existing_3:head by client.4639.0:1 2024-09-13T12:05:05.733761+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-13T12:07:28.575 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-09-13T12:07:28.575 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 29'3 first_divergent_update: 29'106 last_divergent_update: 29'106 2024-09-13T12:07:28.575 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-09-13T12:07:28.575 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-09-13T12:07:28.575 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 29'106 2024-09-13T12:07:28.575 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 29'106 (29'3) delete 1:8881e6d8:::existing_3:head by client.4639.0:1 2024-09-13T12:05:05.733761+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-13T12:07:28.575 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-09-13T12:07:28.575 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 29'3 <= info.log_tail 29'90 2024-09-13T12:07:28.576 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4627.0:1 2024-09-13T12:05:05.713310+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-13T12:07:28.576 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 20 _merge_object_divergent_entries: keeping 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4627.0:1 2024-09-13T12:05:05.713310+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-13T12:07:28.576 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-13T12:07:28.576 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 29'2 first_divergent_update: 29'102 last_divergent_update: 29'102 2024-09-13T12:07:28.576 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-09-13T12:07:28.576 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-09-13T12:07:28.576 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 29'106 2024-09-13T12:07:28.576 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4627.0:1 2024-09-13T12:05:05.713310+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-13T12:07:28.576 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-09-13T12:07:28.577 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 29'2 <= info.log_tail 29'90 2024-09-13T12:07:28.577 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 29'104 (29'1) modify 1:d858c035:::existing_1:head by client.4635.0:1 2024-09-13T12:05:05.729226+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-13T12:07:28.577 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 20 _merge_object_divergent_entries: keeping 29'104 (29'1) modify 1:d858c035:::existing_1:head by client.4635.0:1 2024-09-13T12:05:05.729226+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-13T12:07:28.577 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-13T12:07:28.577 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 29'1 first_divergent_update: 29'104 last_divergent_update: 29'104 2024-09-13T12:07:28.577 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-09-13T12:07:28.577 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-09-13T12:07:28.577 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 29'106 2024-09-13T12:07:28.577 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 29'104 (29'1) modify 1:d858c035:::existing_1:head by client.4635.0:1 2024-09-13T12:05:05.729226+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-13T12:07:28.578 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-09-13T12:07:28.578 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 29'1 <= info.log_tail 29'90 2024-09-13T12:07:28.578 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 29'101 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4626.0:1 2024-09-13T12:05:05.710330+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-13T12:07:28.578 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 20 _merge_object_divergent_entries: keeping 29'101 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4626.0:1 2024-09-13T12:05:05.710330+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-13T12:07:28.578 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-13T12:07:28.578 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 29'101 last_divergent_update: 29'101 2024-09-13T12:07:28.578 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-09-13T12:07:28.578 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:06:56.604+0000 7fbaba0a4640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-09-13T12:07:28.578 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-09-13T12:07:28.579 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-13T12:07:28.581 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-09-13T12:07:28.582 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-13T12:07:28.586 INFO:tasks.workunit.client.0.smithi032.stdout:success 2024-09-13T12:07:28.587 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2024-09-13T12:07:28.587 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.33242.out 2024-09-13T12:07:28.587 INFO:tasks.workunit.client.0.smithi032.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.33242.out': No such file or directory 2024-09-13T12:07:28.587 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2024-09-13T12:07:28.587 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test 2024-09-13T12:07:28.587 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-13T12:07:29.087 INFO:tasks.workunit.client.0.smithi032.stderr:pool 'test' does not exist 2024-09-13T12:07:29.099 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2024-09-13T12:07:29.100 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:07:29.100 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:07:29.100 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:07:29.100 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:07:29.100 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:07:34.414 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:07:34.414 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-13T12:07:34.414 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-13T12:07:34.414 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T12:07:34.414 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-13T12:07:34.414 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:07:34.414 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:07:34.415 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:07:34.415 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:07:34.415 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:07:34.421 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:07:34.421 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T12:07:34.422 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T12:07:34.422 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T12:07:34.424 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T12:07:34.424 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T12:07:34.424 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T12:07:34.425 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:07:34.425 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T12:07:34.426 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T12:07:34.426 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:07:34.427 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:07:34.428 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T12:07:34.428 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-13T12:07:34.452 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T12:07:34.452 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:07:34.452 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:07:34.452 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.33242 2024-09-13T12:07:34.455 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T12:07:34.455 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T12:07:34.455 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-13T12:07:34.455 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-13T12:07:34.455 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/divergent-priors 2024-09-13T12:07:34.455 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/divergent-priors 2024-09-13T12:07:34.455 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-13T12:07:34.455 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T12:07:34.455 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-13T12:07:34.455 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:07:34.455 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:07:34.456 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:07:34.456 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:07:34.456 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:07:34.457 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:07:34.457 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T12:07:34.457 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T12:07:34.458 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T12:07:34.459 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T12:07:34.459 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T12:07:34.459 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T12:07:34.460 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:07:34.460 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T12:07:34.461 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T12:07:34.461 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:07:34.461 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:07:34.463 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T12:07:34.463 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-13T12:07:34.464 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T12:07:34.464 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:07:34.464 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:07:34.464 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.33242 2024-09-13T12:07:34.465 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T12:07:34.465 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T12:07:34.465 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/divergent-priors 2024-09-13T12:07:34.467 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T12:07:34.467 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:07:34.467 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:07:34.467 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.33242 2024-09-13T12:07:34.469 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T12:07:34.469 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T12:07:34.470 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T12:07:34.470 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-13T12:07:34.470 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2024-09-13T12:07:34.470 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2024-09-13T12:07:34.470 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2024-09-13T12:07:34.470 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2024-09-13T12:07:34.470 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2024-09-13T12:07:34.470 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2024-09-13T12:07:34.471 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2024-09-13T12:07:34.472 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2024-09-13T12:07:34.472 INFO:tasks.workunit.client.0.smithi032.stderr:1 2024-09-13T12:07:34.472 INFO:tasks.workunit.client.0.smithi032.stderr:2' 2024-09-13T12:07:34.472 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2024-09-13T12:07:34.472 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/divergent-priors 2024-09-13T12:07:34.472 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T12:07:34.472 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T12:07:34.472 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T12:07:34.472 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/divergent-priors/a 2024-09-13T12:07:34.472 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-13T12:07:34.956 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T12:07:34.956 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:07:34.957 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:07:34.957 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:07:34.957 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:07:34.957 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:07:34.957 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:07:34.957 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-13T12:07:35.149 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T12:07:35.149 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T12:07:35.150 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T12:07:35.150 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T12:07:35.150 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T12:07:35.151 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T12:07:35.151 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T12:07:35.151 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T12:07:35.151 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T12:07:35.151 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:07:35.151 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:07:35.151 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:07:35.151 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-mon.a.asok 2024-09-13T12:07:35.152 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T12:07:35.152 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.33242/ceph-mon.a.asok config get fsid 2024-09-13T12:07:35.241 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T12:07:35.242 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T12:07:35.242 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T12:07:35.242 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T12:07:35.242 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T12:07:35.242 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T12:07:35.242 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T12:07:35.242 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T12:07:35.243 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:07:35.243 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:07:35.243 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:07:35.243 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-mon.a.asok 2024-09-13T12:07:35.243 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T12:07:35.243 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.33242/ceph-mon.a.asok config get mon_host 2024-09-13T12:07:35.323 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2024-09-13T12:07:35.324 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/divergent-priors 2024-09-13T12:07:35.324 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T12:07:35.324 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T12:07:35.324 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T12:07:35.324 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/divergent-priors/x 2024-09-13T12:07:35.324 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T12:07:36.197 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T12:07:36.197 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:07:36.198 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:07:36.198 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:07:36.198 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:07:36.198 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:07:36.198 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:07:36.198 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T12:07:36.199 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T12:07:36.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-09-13T12:07:36.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2024-09-13T12:07:36.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-13T12:07:36.235 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T12:07:36.235 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T12:07:36.235 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T12:07:36.235 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/0 2024-09-13T12:07:36.235 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T12:07:36.235 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:07:36.235 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:07:36.235 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:07:36.235 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-13T12:07:36.235 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-13T12:07:36.236 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T12:07:36.236 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T12:07:36.236 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T12:07:36.236 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T12:07:36.236 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:07:36.236 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:07:36.236 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:07:36.236 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:07:36.236 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:07:36.236 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:07:36.236 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:07:36.237 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:07:36.237 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T12:07:36.237 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T12:07:36.237 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T12:07:36.237 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T12:07:36.237 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:07:36.237 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:07:36.237 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:07:36.237 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:07:36.237 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T12:07:36.237 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T12:07:36.237 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/0 2024-09-13T12:07:36.237 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T12:07:36.238 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=10a21bfc-e16b-4a73-8068-b8ae59576a72 2024-09-13T12:07:36.238 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 10a21bfc-e16b-4a73-8068-b8ae59576a72' 2024-09-13T12:07:36.238 INFO:tasks.workunit.client.0.smithi032.stdout:add osd0 10a21bfc-e16b-4a73-8068-b8ae59576a72 2024-09-13T12:07:36.239 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T12:07:36.260 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAIK+RmH3ZzDxAAXY98Oyh9wwkNcz583BYElg== 2024-09-13T12:07:36.260 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAIK+RmH3ZzDxAAXY98Oyh9wwkNcz583BYElg=="}' 2024-09-13T12:07:36.261 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 10a21bfc-e16b-4a73-8068-b8ae59576a72 -i td/divergent-priors/0/new.json 2024-09-13T12:07:36.539 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:07:36.549 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/0/new.json 2024-09-13T12:07:36.550 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAIK+RmH3ZzDxAAXY98Oyh9wwkNcz583BYElg== --osd-uuid 10a21bfc-e16b-4a73-8068-b8ae59576a72 2024-09-13T12:07:36.588 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:07:36.587+0000 7f1bdbe0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:07:36.608 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:07:36.607+0000 7f1bdbe0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:07:36.609 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:07:36.609+0000 7f1bdbe0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:07:36.610 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:07:36.609+0000 7f1bdbe0e640 -1 bdev(0x562a0591fc00 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-09-13T12:07:36.610 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:07:36.609+0000 7f1bdbe0e640 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-13T12:07:42.575 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-13T12:07:42.575 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T12:07:42.576 INFO:tasks.workunit.client.0.smithi032.stdout:adding osd0 key to auth repository 2024-09-13T12:07:42.576 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T12:07:42.576 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T12:07:43.144 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.0 2024-09-13T12:07:43.144 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T12:07:43.144 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:07:43.144 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T12:07:43.145 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T12:07:43.145 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T12:07:43.180 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:07:43.179+0000 7f6a9eaf1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:07:43.182 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:07:43.181+0000 7f6a9eaf1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:07:43.183 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:07:43.182+0000 7f6a9eaf1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:07:43.611 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T12:07:43.611 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:07:43.611 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T12:07:43.611 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:07:43.611 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:07:43.611 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:07:43.612 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:07:43.612 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:07:43.612 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:07:43.612 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:07:44.044 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:07:44.814 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:07:44.813+0000 7f6a9eaf1640 -1 Falling back to public interface 2024-09-13T12:07:45.045 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:07:45.046 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:07:45.046 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:07:45.046 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:07:45.047 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:07:45.047 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:07:45.464 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:07:46.466 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:07:46.466 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:07:46.466 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:07:46.467 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:07:46.467 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:07:46.467 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:07:46.921 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:07:47.601 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:07:47.600+0000 7f6a9eaf1640 -1 osd.0 0 log_to_monitors true 2024-09-13T12:07:47.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:07:47.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:07:47.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:07:47.923 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:07:47.924 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:07:47.924 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:07:48.351 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:07:49.353 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:07:49.353 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:07:49.353 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:07:49.354 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:07:49.354 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:07:49.354 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:07:49.797 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:07:50.798 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:07:50.798 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:07:50.798 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T12:07:50.799 INFO:tasks.workunit.client.0.smithi032.stdout:5 2024-09-13T12:07:50.800 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:07:50.800 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:07:51.358 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:07:52.360 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:07:52.360 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:07:52.360 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T12:07:52.361 INFO:tasks.workunit.client.0.smithi032.stdout:6 2024-09-13T12:07:52.361 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:07:52.362 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:07:52.781 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:07:53.782 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:07:53.782 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:07:53.782 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T12:07:53.783 INFO:tasks.workunit.client.0.smithi032.stdout:7 2024-09-13T12:07:53.783 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:07:53.783 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:07:54.199 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:07:55.201 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:07:55.201 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:07:55.201 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T12:07:55.201 INFO:tasks.workunit.client.0.smithi032.stdout:8 2024-09-13T12:07:55.202 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:07:55.202 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:07:55.687 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:07:56.688 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:07:56.688 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:07:56.689 INFO:tasks.workunit.client.0.smithi032.stdout:9 2024-09-13T12:07:56.689 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T12:07:56.690 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:07:56.690 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:07:57.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:07:58.140 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:07:58.140 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:07:58.141 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T12:07:58.141 INFO:tasks.workunit.client.0.smithi032.stdout:10 2024-09-13T12:07:58.141 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:07:58.141 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:07:58.705 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:07:59.706 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:07:59.706 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:07:59.706 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T12:07:59.707 INFO:tasks.workunit.client.0.smithi032.stdout:11 2024-09-13T12:07:59.707 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:07:59.707 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:08:00.330 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:08:01.332 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:08:01.333 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:08:01.333 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T12:08:01.333 INFO:tasks.workunit.client.0.smithi032.stdout:12 2024-09-13T12:08:01.333 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:08:01.334 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:08:01.950 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:08:02.951 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:08:02.951 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:08:02.951 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T12:08:02.952 INFO:tasks.workunit.client.0.smithi032.stdout:13 2024-09-13T12:08:02.953 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:08:02.953 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:08:03.669 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:08:04.670 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:08:04.671 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:08:04.671 INFO:tasks.workunit.client.0.smithi032.stdout:14 2024-09-13T12:08:04.671 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T12:08:04.671 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:08:04.672 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:08:05.454 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:08:06.456 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:08:06.456 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:08:06.456 INFO:tasks.workunit.client.0.smithi032.stdout:15 2024-09-13T12:08:06.456 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T12:08:06.457 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:08:06.457 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:08:06.879 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:08:07.880 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:08:07.881 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:08:07.881 INFO:tasks.workunit.client.0.smithi032.stdout:16 2024-09-13T12:08:07.881 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T12:08:07.882 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:08:07.882 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:08:08.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:08:09.735 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:08:09.736 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:08:09.736 INFO:tasks.workunit.client.0.smithi032.stdout:17 2024-09-13T12:08:09.736 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T12:08:09.737 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:08:09.737 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:08:10.204 INFO:tasks.workunit.client.0.smithi032.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4063036085,v1:127.0.0.1:6803/4063036085] [v2:127.0.0.1:6804/4063036085,v1:127.0.0.1:6805/4063036085] exists,up 10a21bfc-e16b-4a73-8068-b8ae59576a72 2024-09-13T12:08:10.209 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:08:10.209 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:08:10.209 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:08:10.209 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-09-13T12:08:10.210 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2024-09-13T12:08:10.210 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-13T12:08:10.210 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T12:08:10.210 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T12:08:10.210 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T12:08:10.210 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/1 2024-09-13T12:08:10.210 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T12:08:10.210 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:08:10.210 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:08:10.210 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:08:10.210 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-13T12:08:10.211 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-13T12:08:10.211 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T12:08:10.211 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T12:08:10.211 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T12:08:10.211 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T12:08:10.211 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:08:10.211 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:08:10.211 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:08:10.211 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:08:10.211 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:08:10.212 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:08:10.212 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:08:10.212 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:08:10.212 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T12:08:10.212 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T12:08:10.212 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T12:08:10.213 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T12:08:10.213 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:08:10.213 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:08:10.214 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:08:10.214 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:08:10.214 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T12:08:10.214 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T12:08:10.214 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/1 2024-09-13T12:08:10.214 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T12:08:10.214 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f377d2cc-d182-4459-bd79-b23c9379a371 2024-09-13T12:08:10.214 INFO:tasks.workunit.client.0.smithi032.stdout:add osd1 f377d2cc-d182-4459-bd79-b23c9379a371 2024-09-13T12:08:10.215 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 f377d2cc-d182-4459-bd79-b23c9379a371' 2024-09-13T12:08:10.215 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T12:08:10.233 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAqK+RmHcPRDRAAV6kcase4IxgyIXqX8I34jQ== 2024-09-13T12:08:10.233 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAqK+RmHcPRDRAAV6kcase4IxgyIXqX8I34jQ=="}' 2024-09-13T12:08:10.233 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f377d2cc-d182-4459-bd79-b23c9379a371 -i td/divergent-priors/1/new.json 2024-09-13T12:08:10.843 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:08:10.854 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/1/new.json 2024-09-13T12:08:10.855 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAqK+RmHcPRDRAAV6kcase4IxgyIXqX8I34jQ== --osd-uuid f377d2cc-d182-4459-bd79-b23c9379a371 2024-09-13T12:08:10.893 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:08:10.891+0000 7f3fb6033640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:08:10.901 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:08:10.899+0000 7f3fb6033640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:08:10.902 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:08:10.900+0000 7f3fb6033640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:08:10.902 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:08:10.901+0000 7f3fb6033640 -1 bdev(0x557fe46ffc00 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-09-13T12:08:10.903 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:08:10.901+0000 7f3fb6033640 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-13T12:08:16.852 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-13T12:08:16.852 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T12:08:16.852 INFO:tasks.workunit.client.0.smithi032.stdout:adding osd1 key to auth repository 2024-09-13T12:08:16.853 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T12:08:16.853 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T12:08:17.425 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.1 2024-09-13T12:08:17.425 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T12:08:17.426 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:08:17.426 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T12:08:17.426 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T12:08:17.426 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T12:08:17.461 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:08:17.459+0000 7f1177c44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:08:17.466 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:08:17.464+0000 7f1177c44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:08:17.467 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:08:17.466+0000 7f1177c44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:08:17.854 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T12:08:17.855 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:08:17.855 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T12:08:17.855 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:08:17.855 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:08:17.855 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:08:17.855 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:08:17.855 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:08:17.856 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:08:17.856 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:08:18.263 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:08:18.828 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:08:18.827+0000 7f1177c44640 -1 Falling back to public interface 2024-09-13T12:08:19.264 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:08:19.264 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:08:19.265 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:08:19.265 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:08:19.265 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:08:19.265 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:08:19.742 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:08:20.743 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:08:20.744 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:08:20.744 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:08:20.744 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:08:20.744 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:08:20.745 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:08:21.193 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:08:21.889 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:08:21.888+0000 7f1177c44640 -1 osd.1 0 log_to_monitors true 2024-09-13T12:08:22.194 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:08:22.194 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:08:22.195 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:08:22.195 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:08:22.195 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:08:22.195 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:08:22.757 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:08:23.758 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:08:23.758 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:08:23.759 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:08:23.759 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:08:23.759 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:08:23.759 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:08:24.180 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:08:25.182 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:08:25.182 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:08:25.182 INFO:tasks.workunit.client.0.smithi032.stdout:5 2024-09-13T12:08:25.183 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T12:08:25.183 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:08:25.183 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:08:25.746 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:08:26.747 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:08:26.747 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:08:26.747 INFO:tasks.workunit.client.0.smithi032.stdout:6 2024-09-13T12:08:26.748 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T12:08:26.748 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:08:26.748 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:08:27.169 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:08:28.170 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:08:28.170 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:08:28.170 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T12:08:28.171 INFO:tasks.workunit.client.0.smithi032.stdout:7 2024-09-13T12:08:28.171 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:08:28.171 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:08:28.641 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:08:29.643 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:08:29.643 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:08:29.643 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T12:08:29.643 INFO:tasks.workunit.client.0.smithi032.stdout:8 2024-09-13T12:08:29.644 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:08:29.644 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:08:30.104 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:08:31.106 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:08:31.107 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:08:31.107 INFO:tasks.workunit.client.0.smithi032.stdout:9 2024-09-13T12:08:31.107 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T12:08:31.107 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:08:31.108 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:08:31.560 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:08:32.561 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:08:32.561 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:08:32.562 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T12:08:32.562 INFO:tasks.workunit.client.0.smithi032.stdout:10 2024-09-13T12:08:32.562 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:08:32.562 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:08:33.113 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:08:34.114 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:08:34.114 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:08:34.114 INFO:tasks.workunit.client.0.smithi032.stdout:11 2024-09-13T12:08:34.115 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T12:08:34.115 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:08:34.115 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:08:34.663 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:08:35.664 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:08:35.664 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:08:35.664 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T12:08:35.665 INFO:tasks.workunit.client.0.smithi032.stdout:12 2024-09-13T12:08:35.665 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:08:35.665 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:08:36.986 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:08:37.987 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:08:37.987 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:08:37.988 INFO:tasks.workunit.client.0.smithi032.stdout:13 2024-09-13T12:08:37.988 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T12:08:37.988 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:08:37.988 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:08:38.811 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:08:39.812 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:08:39.812 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:08:39.812 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T12:08:39.813 INFO:tasks.workunit.client.0.smithi032.stdout:14 2024-09-13T12:08:39.813 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:08:39.813 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:08:40.689 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:08:41.692 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:08:41.693 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:08:41.693 INFO:tasks.workunit.client.0.smithi032.stdout:15 2024-09-13T12:08:41.693 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T12:08:41.694 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:08:41.694 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:08:42.555 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:08:43.557 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:08:43.557 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:08:43.557 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T12:08:43.557 INFO:tasks.workunit.client.0.smithi032.stdout:16 2024-09-13T12:08:43.558 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:08:43.558 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:08:44.382 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:08:45.383 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:08:45.383 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:08:45.384 INFO:tasks.workunit.client.0.smithi032.stdout:17 2024-09-13T12:08:45.384 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T12:08:45.384 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:08:45.384 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:08:45.869 INFO:tasks.workunit.client.0.smithi032.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1424248451,v1:127.0.0.1:6811/1424248451] [v2:127.0.0.1:6812/1424248451,v1:127.0.0.1:6813/1424248451] exists,up f377d2cc-d182-4459-bd79-b23c9379a371 2024-09-13T12:08:45.873 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:08:45.873 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:08:45.873 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:08:45.874 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-09-13T12:08:45.874 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2024-09-13T12:08:45.874 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-13T12:08:45.874 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T12:08:45.874 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T12:08:45.874 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T12:08:45.874 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/2 2024-09-13T12:08:45.874 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T12:08:45.875 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:08:45.875 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:08:45.875 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:08:45.875 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-13T12:08:45.875 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-13T12:08:45.875 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T12:08:45.875 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T12:08:45.875 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T12:08:45.875 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T12:08:45.875 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:08:45.875 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:08:45.875 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:08:45.876 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:08:45.876 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:08:45.876 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:08:45.876 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:08:45.876 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:08:45.876 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T12:08:45.876 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T12:08:45.876 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T12:08:45.876 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T12:08:45.876 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:08:45.876 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:08:45.876 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:08:45.876 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:08:45.877 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T12:08:45.877 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T12:08:45.877 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/2 2024-09-13T12:08:45.877 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T12:08:45.877 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6ac797ab-1924-4ff8-9169-021a47e84482 2024-09-13T12:08:45.877 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 6ac797ab-1924-4ff8-9169-021a47e84482' 2024-09-13T12:08:45.877 INFO:tasks.workunit.client.0.smithi032.stdout:add osd2 6ac797ab-1924-4ff8-9169-021a47e84482 2024-09-13T12:08:45.877 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T12:08:45.898 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBNK+RmVKV3NRAA88x825NLxMiajAeG+3992g== 2024-09-13T12:08:45.898 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBNK+RmVKV3NRAA88x825NLxMiajAeG+3992g=="}' 2024-09-13T12:08:45.898 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6ac797ab-1924-4ff8-9169-021a47e84482 -i td/divergent-priors/2/new.json 2024-09-13T12:08:46.406 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:08:46.417 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/2/new.json 2024-09-13T12:08:46.418 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBNK+RmVKV3NRAA88x825NLxMiajAeG+3992g== --osd-uuid 6ac797ab-1924-4ff8-9169-021a47e84482 2024-09-13T12:08:46.456 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:08:46.455+0000 7fa488a1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:08:46.488 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:08:46.487+0000 7fa488a1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:08:46.488 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:08:46.488+0000 7fa488a1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:08:46.489 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:08:46.488+0000 7fa488a1a640 -1 bdev(0x55d3e0961c00 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-09-13T12:08:46.489 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:08:46.488+0000 7fa488a1a640 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-13T12:08:52.955 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-13T12:08:52.956 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T12:08:52.956 INFO:tasks.workunit.client.0.smithi032.stdout:adding osd2 key to auth repository 2024-09-13T12:08:52.957 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T12:08:52.957 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T12:08:53.529 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.2 2024-09-13T12:08:53.530 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T12:08:53.530 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:08:53.530 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T12:08:53.530 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T12:08:53.530 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T12:08:53.566 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:08:53.564+0000 7fb72fa16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:08:53.571 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:08:53.570+0000 7fb72fa16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:08:53.573 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:08:53.571+0000 7fb72fa16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:08:53.984 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T12:08:53.985 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:08:53.985 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T12:08:53.985 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:08:53.985 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:08:53.985 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:08:53.985 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:08:53.986 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:08:53.986 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:08:53.986 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:08:54.492 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:08:54.923 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:08:54.921+0000 7fb72fa16640 -1 Falling back to public interface 2024-09-13T12:08:55.494 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:08:55.494 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:08:55.494 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:08:55.495 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:08:55.495 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:08:55.495 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:08:55.954 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:08:56.955 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:08:56.955 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:08:56.956 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:08:56.956 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:08:56.956 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:08:56.956 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:08:57.324 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:08:57.322+0000 7fb72fa16640 -1 osd.2 0 log_to_monitors true 2024-09-13T12:08:57.482 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:08:58.484 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:08:58.484 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:08:58.484 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:08:58.484 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:08:58.485 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:08:58.485 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:08:58.928 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:08:59.929 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:08:59.930 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:08:59.930 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:08:59.930 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:08:59.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:08:59.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:09:00.371 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:09:01.373 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:09:01.373 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:09:01.373 INFO:tasks.workunit.client.0.smithi032.stdout:5 2024-09-13T12:09:01.374 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T12:09:01.374 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:09:01.374 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:09:01.921 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:09:02.922 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:09:02.922 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:09:02.922 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T12:09:02.923 INFO:tasks.workunit.client.0.smithi032.stdout:6 2024-09-13T12:09:02.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:09:02.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:09:03.427 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:09:04.428 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:09:04.428 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:09:04.428 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T12:09:04.428 INFO:tasks.workunit.client.0.smithi032.stdout:7 2024-09-13T12:09:04.429 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:09:04.429 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:09:04.887 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:09:05.888 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:09:05.888 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:09:05.888 INFO:tasks.workunit.client.0.smithi032.stdout:8 2024-09-13T12:09:05.889 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T12:09:05.889 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:09:05.889 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:09:06.362 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:09:07.363 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:09:07.363 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:09:07.363 INFO:tasks.workunit.client.0.smithi032.stdout:9 2024-09-13T12:09:07.364 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T12:09:07.364 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:09:07.364 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:09:07.822 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:09:08.824 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:09:08.824 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:09:08.824 INFO:tasks.workunit.client.0.smithi032.stdout:10 2024-09-13T12:09:08.825 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T12:09:08.825 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:09:08.825 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:09:09.272 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:09:10.273 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:09:10.274 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:09:10.274 INFO:tasks.workunit.client.0.smithi032.stdout:11 2024-09-13T12:09:10.274 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T12:09:10.274 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:09:10.275 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:09:10.859 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:09:11.861 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:09:11.861 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:09:11.861 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T12:09:11.861 INFO:tasks.workunit.client.0.smithi032.stdout:12 2024-09-13T12:09:11.861 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:09:11.862 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:09:12.609 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:09:13.610 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:09:13.611 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:09:13.611 INFO:tasks.workunit.client.0.smithi032.stdout:13 2024-09-13T12:09:13.612 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T12:09:13.612 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:09:13.612 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:09:14.343 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:09:15.344 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:09:15.345 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:09:15.345 INFO:tasks.workunit.client.0.smithi032.stdout:14 2024-09-13T12:09:15.345 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T12:09:15.346 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:09:15.346 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:09:16.081 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:09:17.082 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:09:17.083 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:09:17.083 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T12:09:17.083 INFO:tasks.workunit.client.0.smithi032.stdout:15 2024-09-13T12:09:17.083 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:09:17.083 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:09:17.807 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:09:18.808 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:09:18.809 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:09:18.809 INFO:tasks.workunit.client.0.smithi032.stdout:16 2024-09-13T12:09:18.809 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T12:09:18.809 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:09:18.809 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:09:19.573 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:09:20.574 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:09:20.574 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:09:20.575 INFO:tasks.workunit.client.0.smithi032.stdout:17 2024-09-13T12:09:20.575 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T12:09:20.575 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:09:20.575 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:09:21.000 INFO:tasks.workunit.client.0.smithi032.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2830192541,v1:127.0.0.1:6819/2830192541] [v2:127.0.0.1:6820/2830192541,v1:127.0.0.1:6821/2830192541] exists,up 6ac797ab-1924-4ff8-9169-021a47e84482 2024-09-13T12:09:21.000 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:09:21.000 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:09:21.000 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:09:21.001 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2024-09-13T12:09:21.805 INFO:tasks.workunit.client.0.smithi032.stderr:noout is set 2024-09-13T12:09:21.816 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2024-09-13T12:09:22.652 INFO:tasks.workunit.client.0.smithi032.stderr:noin is set 2024-09-13T12:09:22.663 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2024-09-13T12:09:23.547 INFO:tasks.workunit.client.0.smithi032.stderr:nodown is set 2024-09-13T12:09:23.559 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2024-09-13T12:09:23.559 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-13T12:09:24.233 INFO:tasks.workunit.client.0.smithi032.stderr:pool 'test' already exists 2024-09-13T12:09:24.245 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T12:09:25.246 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2024-09-13T12:09:26.018 INFO:tasks.workunit.client.0.smithi032.stderr:set pool 1 size to 3 2024-09-13T12:09:26.030 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2024-09-13T12:09:26.897 INFO:tasks.workunit.client.0.smithi032.stderr:set pool 1 min_size to 2 2024-09-13T12:09:26.908 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2024-09-13T12:09:27.592 INFO:tasks.workunit.client.0.smithi032.stderr:set pool 1 pg_autoscale_mode to on 2024-09-13T12:09:27.604 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: divergent=-1 2024-09-13T12:09:27.604 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: date +%s 2024-09-13T12:09:27.605 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: start_time=1726229367 2024-09-13T12:09:27.605 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:655: TEST_divergent_3: max_duration=300 2024-09-13T12:09:27.606 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' -1 -le -1 ']' 2024-09-13T12:09:27.606 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: flush_pg_stats 2024-09-13T12:09:27.606 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T12:09:27.606 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T12:09:28.091 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T12:09:28.091 INFO:tasks.workunit.client.0.smithi032.stderr:1 2024-09-13T12:09:28.091 INFO:tasks.workunit.client.0.smithi032.stderr:2' 2024-09-13T12:09:28.091 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T12:09:28.091 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:09:28.091 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T12:09:28.216 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836504 2024-09-13T12:09:28.216 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836504 2024-09-13T12:09:28.216 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504' 2024-09-13T12:09:28.216 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:09:28.216 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T12:09:28.341 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672974 2024-09-13T12:09:28.341 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672974 2024-09-13T12:09:28.341 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672974' 2024-09-13T12:09:28.341 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:09:28.341 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T12:09:28.487 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509447 2024-09-13T12:09:28.487 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509447 2024-09-13T12:09:28.487 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672974 2-64424509447' 2024-09-13T12:09:28.487 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:09:28.487 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836504 2024-09-13T12:09:28.487 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:09:28.487 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T12:09:28.488 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836504 2024-09-13T12:09:28.488 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:09:28.488 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836504 2024-09-13T12:09:28.488 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-13T12:09:28.488 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.0 seq 21474836504 2024-09-13T12:09:28.488 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:09:28.926 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836504 2024-09-13T12:09:28.926 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:09:28.926 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672974 2024-09-13T12:09:28.927 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:09:28.927 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T12:09:28.928 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672974 2024-09-13T12:09:28.928 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:09:28.929 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672974 2024-09-13T12:09:28.929 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-09-13T12:09:28.929 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.1 seq 42949672974 2024-09-13T12:09:28.929 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T12:09:29.386 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672974 -lt 42949672974 2024-09-13T12:09:29.386 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:09:29.386 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509447 2024-09-13T12:09:29.386 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:09:29.387 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T12:09:29.388 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509447 2024-09-13T12:09:29.388 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:09:29.389 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509447 2024-09-13T12:09:29.389 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-09-13T12:09:29.389 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.2 seq 64424509447 2024-09-13T12:09:29.389 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T12:09:29.861 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509445 -lt 64424509447 2024-09-13T12:09:29.861 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T12:09:30.863 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T12:09:30.863 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T12:09:31.314 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509447 -lt 64424509447 2024-09-13T12:09:31.315 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: wait_for_clean 2024-09-13T12:09:31.315 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T12:09:31.315 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T12:09:31.315 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T12:09:31.315 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T12:09:31.315 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T12:09:31.315 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T12:09:31.315 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T12:09:31.315 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T12:09:31.315 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T12:09:31.407 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T12:09:31.407 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T12:09:31.407 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T12:09:31.407 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T12:09:31.407 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T12:09:31.407 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T12:09:31.891 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T12:09:31.891 INFO:tasks.workunit.client.0.smithi032.stderr:1 2024-09-13T12:09:31.891 INFO:tasks.workunit.client.0.smithi032.stderr:2' 2024-09-13T12:09:31.891 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T12:09:31.891 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:09:31.891 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T12:09:32.016 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836507 2024-09-13T12:09:32.016 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836507 2024-09-13T12:09:32.016 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507' 2024-09-13T12:09:32.016 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:09:32.016 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T12:09:32.141 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672977 2024-09-13T12:09:32.141 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672977 2024-09-13T12:09:32.141 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-42949672977' 2024-09-13T12:09:32.141 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:09:32.141 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T12:09:32.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509450 2024-09-13T12:09:32.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509450 2024-09-13T12:09:32.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-42949672977 2-64424509450' 2024-09-13T12:09:32.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:09:32.266 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836507 2024-09-13T12:09:32.266 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:09:32.267 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T12:09:32.267 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836507 2024-09-13T12:09:32.267 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:09:32.268 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836507 2024-09-13T12:09:32.268 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-09-13T12:09:32.269 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.0 seq 21474836507 2024-09-13T12:09:32.269 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:09:32.675 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836507 -lt 21474836507 2024-09-13T12:09:32.675 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:09:32.676 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672977 2024-09-13T12:09:32.676 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:09:32.677 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T12:09:32.677 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672977 2024-09-13T12:09:32.677 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:09:32.678 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672977 2024-09-13T12:09:32.678 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-13T12:09:32.679 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.1 seq 42949672977 2024-09-13T12:09:32.679 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T12:09:33.090 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672977 -lt 42949672977 2024-09-13T12:09:33.090 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:09:33.090 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509450 2024-09-13T12:09:33.090 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:09:33.091 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T12:09:33.092 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509450 2024-09-13T12:09:33.092 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:09:33.093 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509450 2024-09-13T12:09:33.093 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.2 seq 64424509450 2024-09-13T12:09:33.093 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-09-13T12:09:33.093 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T12:09:33.550 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509450 -lt 64424509450 2024-09-13T12:09:33.551 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T12:09:33.551 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:09:33.551 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:09:34.111 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T12:09:34.111 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:09:34.111 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:09:34.111 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:09:34.111 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:09:34.111 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:09:34.111 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:09:34.112 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:09:34.522 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T12:09:34.523 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:09:34.523 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:09:34.523 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:09:35.076 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T12:09:35.076 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T12:09:35.076 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T12:09:35.076 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: ceph pg dump pgs --format=json 2024-09-13T12:09:35.076 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2024-09-13T12:09:35.482 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:09:35.495 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: divergent=1 2024-09-13T12:09:35.495 INFO:tasks.workunit.client.0.smithi032.stdout:primary and soon to be divergent is 1 2024-09-13T12:09:35.496 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2024-09-13T12:09:35.496 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:665: TEST_divergent_3: ceph pg dump pgs 2024-09-13T12:09:35.885 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:09:35.885 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T12:09:25.381130+0000 0'0 31:24 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-13T12:09:23.940032+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T20:14:59.072580+0000 0 0 2024-09-13T12:09:35.886 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:09:35.886 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:09:35.886 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:09:35.897 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: date +%s 2024-09-13T12:09:35.897 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: current_time=1726229375 2024-09-13T12:09:35.898 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: expr 1726229375 - 1726229367 2024-09-13T12:09:35.900 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: elapsed_time=8 2024-09-13T12:09:35.900 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: '[' 8 -gt 300 ']' 2024-09-13T12:09:35.900 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' 1 -le -1 ']' 2024-09-13T12:09:35.900 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:675: TEST_divergent_3: local non_divergent= 2024-09-13T12:09:35.900 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-09-13T12:09:35.901 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 0 = 1 ']' 2024-09-13T12:09:35.901 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 0' 2024-09-13T12:09:35.901 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-09-13T12:09:35.901 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 1 = 1 ']' 2024-09-13T12:09:35.901 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:679: TEST_divergent_3: continue 2024-09-13T12:09:35.901 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-09-13T12:09:35.901 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 2 = 1 ']' 2024-09-13T12:09:35.901 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 0 2' 2024-09-13T12:09:35.901 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:684: TEST_divergent_3: echo 'writing initial objects' 2024-09-13T12:09:35.901 INFO:tasks.workunit.client.0.smithi032.stdout:writing initial objects 2024-09-13T12:09:35.902 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: seq 1 100 2024-09-13T12:09:35.902 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:35.902 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2024-09-13T12:09:36.072 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:36.072 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2024-09-13T12:09:36.223 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:36.223 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2024-09-13T12:09:36.378 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:36.378 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2024-09-13T12:09:36.576 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:36.576 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2024-09-13T12:09:36.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:36.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2024-09-13T12:09:36.880 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:36.880 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2024-09-13T12:09:37.013 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:37.013 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2024-09-13T12:09:37.156 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:37.156 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2024-09-13T12:09:37.298 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:37.298 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2024-09-13T12:09:37.441 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:37.441 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2024-09-13T12:09:37.584 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:37.584 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2024-09-13T12:09:37.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:37.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2024-09-13T12:09:37.877 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:37.877 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2024-09-13T12:09:38.019 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:38.019 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2024-09-13T12:09:38.170 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:38.170 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2024-09-13T12:09:38.321 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:38.321 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2024-09-13T12:09:38.480 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:38.481 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2024-09-13T12:09:38.623 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:38.623 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2024-09-13T12:09:38.757 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:38.757 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2024-09-13T12:09:38.891 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:38.891 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2024-09-13T12:09:39.025 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:39.025 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2024-09-13T12:09:39.176 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:39.176 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2024-09-13T12:09:39.328 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:39.328 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2024-09-13T12:09:39.494 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:39.495 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2024-09-13T12:09:39.637 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:39.637 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2024-09-13T12:09:39.788 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:39.788 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2024-09-13T12:09:39.922 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:39.922 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2024-09-13T12:09:40.064 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:40.064 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2024-09-13T12:09:40.198 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:40.199 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2024-09-13T12:09:40.332 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:40.333 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2024-09-13T12:09:40.483 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:40.484 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2024-09-13T12:09:40.869 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:40.869 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2024-09-13T12:09:41.021 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:41.021 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2024-09-13T12:09:41.188 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:41.188 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2024-09-13T12:09:41.347 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:41.347 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2024-09-13T12:09:41.524 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:41.525 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2024-09-13T12:09:41.657 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:41.658 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2024-09-13T12:09:41.791 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:41.791 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2024-09-13T12:09:41.934 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:41.934 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2024-09-13T12:09:42.076 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:42.076 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2024-09-13T12:09:42.235 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:42.235 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2024-09-13T12:09:42.386 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:42.386 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2024-09-13T12:09:42.570 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:42.570 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2024-09-13T12:09:43.848 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:43.848 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2024-09-13T12:09:44.006 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:44.006 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2024-09-13T12:09:44.198 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:44.198 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2024-09-13T12:09:44.340 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:44.341 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2024-09-13T12:09:44.500 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:44.500 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2024-09-13T12:09:44.655 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:44.655 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2024-09-13T12:09:44.877 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:44.877 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2024-09-13T12:09:45.036 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:45.036 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2024-09-13T12:09:45.312 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:45.313 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2024-09-13T12:09:45.472 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:45.473 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2024-09-13T12:09:45.623 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:45.623 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2024-09-13T12:09:45.866 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:45.866 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2024-09-13T12:09:46.025 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:46.026 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2024-09-13T12:09:46.193 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:46.193 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2024-09-13T12:09:46.336 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:46.336 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2024-09-13T12:09:46.503 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:46.504 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2024-09-13T12:09:46.679 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:46.679 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2024-09-13T12:09:47.988 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:47.988 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2024-09-13T12:09:48.173 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:48.173 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2024-09-13T12:09:48.324 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:48.324 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2024-09-13T12:09:48.483 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:48.483 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2024-09-13T12:09:48.684 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:48.685 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2024-09-13T12:09:49.214 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:49.214 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2024-09-13T12:09:49.608 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:49.608 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2024-09-13T12:09:49.803 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:49.803 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2024-09-13T12:09:50.089 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:50.090 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2024-09-13T12:09:50.270 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:50.270 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2024-09-13T12:09:50.730 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:50.731 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2024-09-13T12:09:50.981 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:50.981 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2024-09-13T12:09:51.182 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:51.183 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2024-09-13T12:09:51.527 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:51.527 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2024-09-13T12:09:51.820 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:51.820 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2024-09-13T12:09:52.304 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:52.304 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2024-09-13T12:09:52.550 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:52.550 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2024-09-13T12:09:52.761 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:52.761 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2024-09-13T12:09:53.221 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:53.221 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2024-09-13T12:09:53.424 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:53.424 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2024-09-13T12:09:53.716 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:53.716 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2024-09-13T12:09:53.985 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:53.985 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2024-09-13T12:09:54.245 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:54.245 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2024-09-13T12:09:54.405 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:54.405 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2024-09-13T12:09:54.698 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:54.698 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2024-09-13T12:09:54.894 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:54.894 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2024-09-13T12:09:55.251 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:55.252 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2024-09-13T12:09:55.554 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:55.554 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2024-09-13T12:09:55.722 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:55.722 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2024-09-13T12:09:55.948 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:55.949 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2024-09-13T12:09:56.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:56.267 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2024-09-13T12:09:56.426 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:56.426 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2024-09-13T12:09:56.661 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:56.661 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2024-09-13T12:09:57.021 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:57.022 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2024-09-13T12:09:57.461 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:57.461 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2024-09-13T12:09:57.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:57.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2024-09-13T12:09:57.986 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:57.986 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2024-09-13T12:09:58.137 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:58.137 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2024-09-13T12:09:58.288 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-13T12:09:58.288 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2024-09-13T12:09:58.447 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-13T12:09:58.447 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: wait_for_clean 2024-09-13T12:09:58.447 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T12:09:58.447 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T12:09:58.448 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T12:09:58.448 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-13T12:09:58.448 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T12:09:58.448 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T12:09:58.449 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T12:09:58.449 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T12:09:58.449 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T12:09:58.508 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-13T12:09:58.508 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T12:09:58.508 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T12:09:58.509 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T12:09:58.509 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T12:09:58.509 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T12:09:59.010 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T12:09:59.011 INFO:tasks.workunit.client.0.smithi032.stderr:1 2024-09-13T12:09:59.011 INFO:tasks.workunit.client.0.smithi032.stderr:2' 2024-09-13T12:09:59.011 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T12:09:59.011 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:09:59.011 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T12:09:59.135 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836514 2024-09-13T12:09:59.135 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836514 2024-09-13T12:09:59.136 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836514' 2024-09-13T12:09:59.136 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:09:59.136 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T12:09:59.261 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672984 2024-09-13T12:09:59.261 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672984 2024-09-13T12:09:59.261 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836514 1-42949672984' 2024-09-13T12:09:59.261 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:09:59.261 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T12:09:59.387 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509457 2024-09-13T12:09:59.387 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509457 2024-09-13T12:09:59.387 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836514 1-42949672984 2-64424509457' 2024-09-13T12:09:59.388 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:09:59.388 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836514 2024-09-13T12:09:59.388 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:09:59.389 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T12:09:59.389 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836514 2024-09-13T12:09:59.389 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:09:59.389 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836514 2024-09-13T12:09:59.389 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.0 seq 21474836514 2024-09-13T12:09:59.390 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2024-09-13T12:09:59.390 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:09:59.803 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836512 -lt 21474836514 2024-09-13T12:09:59.803 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T12:10:00.805 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T12:10:00.805 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:10:01.370 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836514 -lt 21474836514 2024-09-13T12:10:01.370 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:10:01.371 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672984 2024-09-13T12:10:01.371 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:10:01.371 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T12:10:01.372 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672984 2024-09-13T12:10:01.372 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:10:01.373 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672984 2024-09-13T12:10:01.373 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-09-13T12:10:01.374 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.1 seq 42949672984 2024-09-13T12:10:01.374 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T12:10:01.788 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672984 -lt 42949672984 2024-09-13T12:10:01.788 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:10:01.789 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509457 2024-09-13T12:10:01.789 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:10:01.790 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T12:10:01.790 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509457 2024-09-13T12:10:01.790 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:10:01.791 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509457 2024-09-13T12:10:01.792 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.2 seq 64424509457 2024-09-13T12:10:01.792 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-09-13T12:10:01.792 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T12:10:02.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509457 -lt 64424509457 2024-09-13T12:10:02.267 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T12:10:02.267 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:10:02.267 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:10:03.000 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 32 == 0 2024-09-13T12:10:03.000 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:10:03.000 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:10:03.000 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:10:03.000 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:10:03.000 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:10:03.000 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:10:03.000 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:10:03.537 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=32 2024-09-13T12:10:03.537 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:10:03.537 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:10:03.537 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:10:04.114 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 = 32 2024-09-13T12:10:04.114 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T12:10:04.114 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T12:10:04.114 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: get_pg test existing_1 2024-09-13T12:10:04.114 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=test 2024-09-13T12:10:04.115 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=existing_1 2024-09-13T12:10:04.115 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map test existing_1 2024-09-13T12:10:04.115 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T12:10:04.594 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: local pgid=1.1b 2024-09-13T12:10:04.594 INFO:tasks.workunit.client.0.smithi032.stdout:blackholing osds 0 2 2024-09-13T12:10:04.595 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: echo 'blackholing osds 0 2' 2024-09-13T12:10:04.595 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:697: TEST_divergent_3: ceph pg dump pgs 2024-09-13T12:10:04.996 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:10:04.996 INFO:tasks.workunit.client.0.smithi032.stdout:1.1f 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:09:53.714446+0000 38'45 45:130 [1,2,0] 1 [1,2,0] 1 38'45 2024-09-13T12:09:53.714380+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T18:19:07.204187+0000 3 0 2024-09-13T12:10:04.996 INFO:tasks.workunit.client.0.smithi032.stdout:1.1e 3 0 0 0 0 1615 0 0 15 30 15 active+clean 2024-09-13T12:09:57.574264+0000 44'45 45:103 [1,0,2] 1 [1,0,2] 1 36'44 2024-09-13T12:09:53.952476+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:10:04.996 INFO:tasks.workunit.client.0.smithi032.stdout:1.1d 4 0 0 0 0 1932 0 0 16 30 16 active+clean 2024-09-13T12:09:49.397470+0000 39'46 45:127 [2,0,1] 2 [2,0,1] 2 34'44 2024-09-13T12:09:49.397412+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:06:06.271370+0000 2 0 2024-09-13T12:10:04.996 INFO:tasks.workunit.client.0.smithi032.stdout:1.b 3 0 0 0 0 1449 0 0 14 30 14 active+clean 2024-09-13T12:09:57.774917+0000 42'44 45:116 [1,2,0] 1 [1,2,0] 1 42'44 2024-09-13T12:09:57.774856+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T21:52:48.834212+0000 3 0 2024-09-13T12:10:04.996 INFO:tasks.workunit.client.0.smithi032.stdout:1.a 3 0 0 0 0 1449 0 0 14 30 14 active+clean 2024-09-13T12:09:50.853117+0000 34'44 46:124 [0,1,2] 0 [0,1,2] 0 34'44 2024-09-13T12:09:50.853058+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T22:23:44.907922+0000 3 0 2024-09-13T12:10:04.996 INFO:tasks.workunit.client.0.smithi032.stdout:1.9 5 0 0 0 0 2415 0 0 15 30 15 active+clean 2024-09-13T12:09:55.340956+0000 42'45 45:115 [2,0,1] 2 [2,0,1] 2 41'44 2024-09-13T12:09:55.340894+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T16:59:28.251016+0000 4 0 2024-09-13T12:10:04.996 INFO:tasks.workunit.client.0.smithi032.stdout:1.8 3 0 0 0 0 1449 0 0 16 30 16 active+clean 2024-09-13T12:09:58.652201+0000 42'46 45:122 [1,0,2] 1 [1,0,2] 1 42'46 2024-09-13T12:09:58.652148+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T13:10:05.453890+0000 3 0 2024-09-13T12:10:04.997 INFO:tasks.workunit.client.0.smithi032.stdout:1.7 2 0 0 0 0 966 0 0 14 30 14 active+clean 2024-09-13T12:09:55.719638+0000 34'44 45:116 [1,2,0] 1 [1,2,0] 1 34'44 2024-09-13T12:09:55.719570+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T17:29:02.601805+0000 2 0 2024-09-13T12:10:04.997 INFO:tasks.workunit.client.0.smithi032.stdout:1.6 5 0 0 0 0 2098 0 0 16 30 16 active+clean 2024-09-13T12:09:57.574405+0000 41'46 45:122 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-13T12:09:23.940032+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:10:04.997 INFO:tasks.workunit.client.0.smithi032.stdout:1.5 6 0 0 0 0 2898 0 0 16 30 16 active+clean 2024-09-13T12:09:52.375479+0000 42'46 45:116 [2,0,1] 2 [2,0,1] 2 38'45 2024-09-13T12:09:52.375427+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T21:06:10.272845+0000 5 0 2024-09-13T12:10:04.997 INFO:tasks.workunit.client.0.smithi032.stdout:1.4 2 0 0 0 0 966 0 0 14 30 14 active+clean 2024-09-13T12:09:46.701281+0000 34'44 45:120 [1,0,2] 1 [1,0,2] 1 34'44 2024-09-13T12:09:46.701189+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T17:13:38.494261+0000 2 0 2024-09-13T12:10:04.997 INFO:tasks.workunit.client.0.smithi032.stdout:1.3 5 0 0 0 0 2415 0 0 15 30 15 active+clean 2024-09-13T12:09:52.263095+0000 44'45 45:117 [1,2,0] 1 [1,2,0] 1 32'43 2024-09-13T12:09:44.638103+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T21:28:30.784756+0000 3 0 2024-09-13T12:10:04.997 INFO:tasks.workunit.client.0.smithi032.stdout:1.2 1 0 0 0 0 483 0 0 14 30 14 active+clean 2024-09-13T12:09:55.946647+0000 39'44 46:112 [0,1,2] 0 [0,1,2] 0 39'44 2024-09-13T12:09:55.946588+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:22:06.603899+0000 1 0 2024-09-13T12:10:04.997 INFO:tasks.workunit.client.0.smithi032.stdout:1.c 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:09:51.776961+0000 41'45 45:121 [1,0,2] 1 [1,0,2] 1 34'44 2024-09-13T12:09:51.776896+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T15:11:25.653962+0000 2 0 2024-09-13T12:10:04.997 INFO:tasks.workunit.client.0.smithi032.stdout:1.1 2 0 0 0 0 966 0 0 15 30 15 active+clean 2024-09-13T12:09:48.455915+0000 41'45 45:115 [2,0,1] 2 [2,0,1] 2 32'43 2024-09-13T12:09:48.455862+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:32:03.475174+0000 0 0 2024-09-13T12:10:04.997 INFO:tasks.workunit.client.0.smithi032.stdout:1.1c 2 0 0 0 0 966 0 0 14 30 14 active+clean 2024-09-13T12:09:49.691499+0000 34'44 45:120 [1,0,2] 1 [1,0,2] 1 34'44 2024-09-13T12:09:49.691404+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T22:46:12.917366+0000 2 0 2024-09-13T12:10:04.997 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 3 0 0 0 0 1449 0 0 14 30 14 active+clean 2024-09-13T12:09:43.710429+0000 42'44 45:120 [1,0,2] 1 [1,0,2] 1 32'43 2024-09-13T12:09:43.710370+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:48:46.408051+0000 2 0 2024-09-13T12:10:04.998 INFO:tasks.workunit.client.0.smithi032.stdout:1.d 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:09:50.728238+0000 42'45 45:115 [2,0,1] 2 [2,0,1] 2 32'43 2024-09-13T12:09:50.728150+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T12:34:58.564329+0000 1 0 2024-09-13T12:10:04.998 INFO:tasks.workunit.client.0.smithi032.stdout:1.e 4 0 0 0 0 1932 0 0 15 30 15 active+clean 2024-09-13T12:09:57.574280+0000 42'45 45:115 [1,0,2] 1 [1,0,2] 1 32'43 2024-09-13T12:09:49.907160+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:12:01.339767+0000 2 0 2024-09-13T12:10:04.998 INFO:tasks.workunit.client.0.smithi032.stdout:1.f 4 0 0 0 0 1932 0 0 14 30 14 active+clean 2024-09-13T12:09:59.616237+0000 38'44 45:118 [1,2,0] 1 [1,2,0] 1 38'44 2024-09-13T12:09:59.616191+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T15:52:51.901842+0000 4 0 2024-09-13T12:10:04.998 INFO:tasks.workunit.client.0.smithi032.stdout:1.10 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:09:52.759410+0000 43'45 45:121 [1,0,2] 1 [1,0,2] 1 34'44 2024-09-13T12:09:52.759350+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T18:50:37.130834+0000 2 0 2024-09-13T12:10:04.998 INFO:tasks.workunit.client.0.smithi032.stdout:1.11 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:09:53.421673+0000 34'45 45:126 [2,0,1] 2 [2,0,1] 2 34'45 2024-09-13T12:09:53.421579+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T21:04:04.783312+0000 3 0 2024-09-13T12:10:04.998 INFO:tasks.workunit.client.0.smithi032.stdout:1.12 1 0 0 0 0 483 0 0 14 30 14 active+clean 2024-09-13T12:09:52.858712+0000 34'44 46:123 [0,1,2] 0 [0,1,2] 0 34'44 2024-09-13T12:09:52.858658+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T15:34:21.017146+0000 1 0 2024-09-13T12:10:04.998 INFO:tasks.workunit.client.0.smithi032.stdout:1.13 3 0 0 0 0 1615 0 0 15 30 15 active+clean 2024-09-13T12:09:52.300410+0000 44'45 45:112 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-13T12:09:23.940032+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:10:04.998 INFO:tasks.workunit.client.0.smithi032.stdout:1.14 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:09:50.728046+0000 44'45 45:121 [1,0,2] 1 [1,0,2] 1 32'43 2024-09-13T12:09:50.727967+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T22:52:04.595740+0000 1 0 2024-09-13T12:10:04.998 INFO:tasks.workunit.client.0.smithi032.stdout:1.15 3 0 0 0 0 1449 0 0 14 30 14 active+clean 2024-09-13T12:09:54.403068+0000 38'44 45:114 [2,0,1] 2 [2,0,1] 2 38'44 2024-09-13T12:09:54.402988+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T17:48:13.651473+0000 3 0 2024-09-13T12:10:04.998 INFO:tasks.workunit.client.0.smithi032.stdout:1.16 4 0 0 0 0 2098 0 0 16 30 16 active+clean 2024-09-13T12:09:57.574311+0000 38'46 45:113 [1,0,2] 1 [1,0,2] 1 38'46 2024-09-13T12:09:54.892264+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:10:04.998 INFO:tasks.workunit.client.0.smithi032.stdout:1.17 1 0 0 0 0 483 0 0 13 30 13 active+clean 2024-09-13T12:09:56.659063+0000 32'43 45:115 [1,2,0] 1 [1,2,0] 1 32'43 2024-09-13T12:09:56.659002+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T12:49:40.107974+0000 1 0 2024-09-13T12:10:04.998 INFO:tasks.workunit.client.0.smithi032.stdout:1.18 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:09:47.765579+0000 44'45 45:121 [1,0,2] 1 [1,0,2] 1 32'43 2024-09-13T12:09:47.765518+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T18:03:34.766505+0000 1 0 2024-09-13T12:10:04.998 INFO:tasks.workunit.client.0.smithi032.stdout:1.19 2 0 0 0 0 966 0 0 14 30 14 active+clean 2024-09-13T12:09:51.375393+0000 34'44 45:124 [2,0,1] 2 [2,0,1] 2 34'44 2024-09-13T12:09:51.375338+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T15:02:48.345008+0000 2 0 2024-09-13T12:10:04.999 INFO:tasks.workunit.client.0.smithi032.stdout:1.1a 4 0 0 0 0 1932 0 0 15 30 15 active+clean 2024-09-13T12:09:51.953485+0000 41'45 46:124 [0,1,2] 0 [0,1,2] 0 34'44 2024-09-13T12:09:51.953415+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T21:45:51.945598+0000 3 0 2024-09-13T12:10:04.999 INFO:tasks.workunit.client.0.smithi032.stdout:1.1b 5 0 0 0 0 2415 0 0 16 30 16 active+clean 2024-09-13T12:09:54.677840+0000 44'46 45:126 [1,2,0] 1 [1,2,0] 1 36'45 2024-09-13T12:09:54.677751+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T17:21:10.868211+0000 4 0 2024-09-13T12:10:04.999 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:10:04.999 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:10:04.999 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:10:05.006 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-09-13T12:10:05.007 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.0 2024-09-13T12:10:05.007 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-13T12:10:05.007 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-13T12:10:05.007 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:10:05.008 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:10:05.008 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:10:05.008 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-osd.0.asok 2024-09-13T12:10:05.008 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-09-13T12:10:05.008 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.33242/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-13T12:10:05.088 INFO:tasks.workunit.client.0.smithi032.stdout:{ 2024-09-13T12:10:05.088 INFO:tasks.workunit.client.0.smithi032.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-13T12:10:05.088 INFO:tasks.workunit.client.0.smithi032.stdout:} 2024-09-13T12:10:05.095 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-09-13T12:10:05.096 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.2 2024-09-13T12:10:05.097 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-13T12:10:05.097 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-13T12:10:05.097 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:10:05.097 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:10:05.097 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:10:05.097 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-osd.2.asok 2024-09-13T12:10:05.097 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-09-13T12:10:05.097 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.33242/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-13T12:10:05.179 INFO:tasks.workunit.client.0.smithi032.stdout:{ 2024-09-13T12:10:05.179 INFO:tasks.workunit.client.0.smithi032.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-13T12:10:05.179 INFO:tasks.workunit.client.0.smithi032.stdout:} 2024-09-13T12:10:05.186 INFO:tasks.workunit.client.0.smithi032.stdout:create new divergent objects 2024-09-13T12:10:05.186 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: echo 'create new divergent objects' 2024-09-13T12:10:05.186 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: seq 1 2 2024-09-13T12:10:05.187 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-13T12:10:05.187 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-13T12:10:05.187 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_1 2024-09-13T12:10:05.188 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:710: TEST_divergent_3: echo 'writing divergent objects' 2024-09-13T12:10:05.188 INFO:tasks.workunit.client.0.smithi032.stdout:writing divergent objects 2024-09-13T12:10:05.189 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_2 2024-09-13T12:10:05.189 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: seq 1 2 2024-09-13T12:10:05.189 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-13T12:10:05.189 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-13T12:10:05.189 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2024-09-13T12:10:05.189 INFO:tasks.workunit.client.0.smithi032.stdout:remove divergent objects 2024-09-13T12:10:05.190 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: echo 'remove divergent objects' 2024-09-13T12:10:05.190 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2024-09-13T12:10:05.190 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: seq 1 2 2024-09-13T12:10:05.190 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-13T12:10:05.191 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 1 + 2 2024-09-13T12:10:05.193 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=3 2024-09-13T12:10:05.193 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-13T12:10:05.193 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_3 2024-09-13T12:10:05.193 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 2 + 2 2024-09-13T12:10:05.194 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=4 2024-09-13T12:10:05.195 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:722: TEST_divergent_3: sleep 10 2024-09-13T12:10:05.195 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_4 2024-09-13T12:10:15.197 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:723: TEST_divergent_3: killall -9 rados 2024-09-13T12:10:15.200 INFO:tasks.workunit.client.0.smithi032.stderr:rados: no process found 2024-09-13T12:10:15.200 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: echo 'killing all the osds' 2024-09-13T12:10:15.201 INFO:tasks.workunit.client.0.smithi032.stdout:killing all the osds 2024-09-13T12:10:15.201 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:727: TEST_divergent_3: ceph pg dump pgs 2024-09-13T12:10:15.600 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:10:15.601 INFO:tasks.workunit.client.0.smithi032.stdout:1.1f 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:09:53.714446+0000 38'45 54:153 [1,2,0] 1 [1,2,0] 1 38'45 2024-09-13T12:09:53.714380+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:49:50.581373+0000 3 0 2024-09-13T12:10:15.601 INFO:tasks.workunit.client.0.smithi032.stdout:1.1e 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:10:01.695484+0000 44'45 54:134 [1,0,2] 1 [1,0,2] 1 44'45 2024-09-13T12:10:01.695428+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T12:22:38.123714+0000 3 0 2024-09-13T12:10:15.601 INFO:tasks.workunit.client.0.smithi032.stdout:1.1d 4 0 0 0 0 1932 0 0 16 30 16 activating 2024-09-13T12:10:10.851143+0000 39'46 56:154 [2,1,0] 2 [2,1,0] 2 34'44 2024-09-13T12:09:49.397412+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 no scrub is scheduled 2 0 2024-09-13T12:10:15.601 INFO:tasks.workunit.client.0.smithi032.stdout:1.b 3 0 0 0 0 1449 0 0 14 30 14 active+clean 2024-09-13T12:10:07.059118+0000 42'44 54:145 [1,0,2] 1 [1,0,2] 1 42'44 2024-09-13T12:09:57.774856+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T17:13:44.818064+0000 3 0 2024-09-13T12:10:15.601 INFO:tasks.workunit.client.0.smithi032.stdout:1.a 3 0 0 0 0 1449 0 0 14 30 14 active+clean 2024-09-13T12:10:06.700195+0000 34'44 54:115 [1,2,0] 1 [1,2,0] 1 34'44 2024-09-13T12:10:06.700135+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T23:32:12.380343+0000 3 0 2024-09-13T12:10:15.601 INFO:tasks.workunit.client.0.smithi032.stdout:1.9 5 0 0 0 0 2086 0 0 16 30 16 active+clean 2024-09-13T12:10:03.691468+0000 51'46 54:139 [1,2,0] 1 [1,2,0] 1 42'45 2024-09-13T12:10:03.691408+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T14:28:06.215769+0000 5 0 2024-09-13T12:10:15.601 INFO:tasks.workunit.client.0.smithi032.stdout:1.8 3 0 0 0 0 1449 0 0 16 30 16 active+clean 2024-09-13T12:10:01.491569+0000 42'46 54:151 [1,2,0] 1 [1,2,0] 1 42'46 2024-09-13T12:09:58.652148+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T12:26:41.295513+0000 3 0 2024-09-13T12:10:15.601 INFO:tasks.workunit.client.0.smithi032.stdout:1.7 2 0 0 0 0 966 0 0 14 30 14 active+clean 2024-09-13T12:09:55.719638+0000 34'44 54:139 [1,2,0] 1 [1,2,0] 1 34'44 2024-09-13T12:09:55.719570+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T14:02:28.963772+0000 2 0 2024-09-13T12:10:15.601 INFO:tasks.workunit.client.0.smithi032.stdout:1.6 4 0 0 0 0 1932 0 0 16 30 16 active+clean 2024-09-13T12:10:02.648923+0000 41'46 54:153 [1,0,2] 1 [1,0,2] 1 41'46 2024-09-13T12:10:02.648875+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T15:01:46.823265+0000 4 0 2024-09-13T12:10:15.601 INFO:tasks.workunit.client.0.smithi032.stdout:1.5 6 0 0 0 0 2898 0 0 16 30 16 active+clean 2024-09-13T12:09:52.375479+0000 42'46 56:144 [2,0,1] 2 [2,0,1] 2 38'45 2024-09-13T12:09:52.375427+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T12:21:52.219300+0000 5 0 2024-09-13T12:10:15.602 INFO:tasks.workunit.client.0.smithi032.stdout:1.4 2 0 0 0 0 966 0 0 14 30 14 active+clean 2024-09-13T12:09:46.701281+0000 34'44 54:143 [1,0,2] 1 [1,0,2] 1 34'44 2024-09-13T12:09:46.701189+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T21:17:18.496702+0000 2 0 2024-09-13T12:10:15.602 INFO:tasks.workunit.client.0.smithi032.stdout:1.3 5 0 0 0 0 2415 0 0 15 30 15 active+clean 2024-09-13T12:09:52.263095+0000 44'45 54:140 [1,2,0] 1 [1,2,0] 1 32'43 2024-09-13T12:09:44.638103+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T12:51:27.106043+0000 3 0 2024-09-13T12:10:15.602 INFO:tasks.workunit.client.0.smithi032.stdout:1.2 1 0 0 0 0 483 0 0 14 30 14 active+clean 2024-09-13T12:09:55.946647+0000 39'44 56:137 [0,1,2] 0 [0,1,2] 0 39'44 2024-09-13T12:09:55.946588+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:35:10.767429+0000 1 0 2024-09-13T12:10:15.602 INFO:tasks.workunit.client.0.smithi032.stdout:1.c 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:09:51.776961+0000 41'45 54:144 [1,0,2] 1 [1,0,2] 1 34'44 2024-09-13T12:09:51.776896+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T22:33:34.079244+0000 2 0 2024-09-13T12:10:15.602 INFO:tasks.workunit.client.0.smithi032.stdout:1.1 2 0 0 0 0 966 0 0 15 30 15 active+clean 2024-09-13T12:09:48.455915+0000 41'45 56:143 [2,0,1] 2 [2,0,1] 2 32'43 2024-09-13T12:09:48.455862+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T14:27:09.423237+0000 0 0 2024-09-13T12:10:15.602 INFO:tasks.workunit.client.0.smithi032.stdout:1.1c 2 0 0 0 0 966 0 0 14 30 14 active+clean 2024-09-13T12:09:49.691499+0000 34'44 54:143 [1,0,2] 1 [1,0,2] 1 34'44 2024-09-13T12:09:49.691404+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T16:37:09.360899+0000 2 0 2024-09-13T12:10:15.602 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 3 0 0 0 0 1449 0 0 14 30 14 active+clean 2024-09-13T12:09:43.710429+0000 42'44 54:143 [1,0,2] 1 [1,0,2] 1 32'43 2024-09-13T12:09:43.710370+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T18:57:43.048737+0000 2 0 2024-09-13T12:10:15.602 INFO:tasks.workunit.client.0.smithi032.stdout:1.d 3 0 0 0 0 1449 0 0 15 30 15 activating 2024-09-13T12:10:10.851135+0000 42'45 56:142 [2,1,0] 2 [2,1,0] 2 32'43 2024-09-13T12:09:50.728150+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 no scrub is scheduled 1 0 2024-09-13T12:10:15.602 INFO:tasks.workunit.client.0.smithi032.stdout:1.e 4 0 0 0 0 1449 0 0 17 30 17 active+clean 2024-09-13T12:09:57.574280+0000 51'47 54:140 [1,0,2] 1 [1,0,2] 1 32'43 2024-09-13T12:09:49.907160+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:17:29.233183+0000 2 0 2024-09-13T12:10:15.602 INFO:tasks.workunit.client.0.smithi032.stdout:1.f 4 0 0 0 0 1932 0 0 14 30 14 active+clean 2024-09-13T12:09:59.616237+0000 38'44 54:141 [1,2,0] 1 [1,2,0] 1 38'44 2024-09-13T12:09:59.616191+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T19:29:23.812470+0000 4 0 2024-09-13T12:10:15.602 INFO:tasks.workunit.client.0.smithi032.stdout:1.10 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:09:52.759410+0000 43'45 54:144 [1,0,2] 1 [1,0,2] 1 34'44 2024-09-13T12:09:52.759350+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T14:31:33.259233+0000 2 0 2024-09-13T12:10:15.602 INFO:tasks.workunit.client.0.smithi032.stdout:1.11 2 0 0 0 0 966 0 0 16 30 16 active+clean 2024-09-13T12:09:53.421673+0000 51'46 56:155 [2,0,1] 2 [2,0,1] 2 34'45 2024-09-13T12:09:53.421579+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T12:40:23.459149+0000 3 0 2024-09-13T12:10:15.602 INFO:tasks.workunit.client.0.smithi032.stdout:1.12 1 0 0 0 0 483 0 0 14 30 14 active+clean 2024-09-13T12:09:52.858712+0000 34'44 56:148 [0,1,2] 0 [0,1,2] 0 34'44 2024-09-13T12:09:52.858658+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T17:10:31.364184+0000 1 0 2024-09-13T12:10:15.602 INFO:tasks.workunit.client.0.smithi032.stdout:1.13 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:10:04.717979+0000 44'45 54:142 [1,2,0] 1 [1,2,0] 1 44'45 2024-09-13T12:10:04.717933+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T18:46:14.412995+0000 3 0 2024-09-13T12:10:15.603 INFO:tasks.workunit.client.0.smithi032.stdout:1.14 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:09:50.728046+0000 44'45 54:144 [1,0,2] 1 [1,0,2] 1 32'43 2024-09-13T12:09:50.727967+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T14:51:19.255424+0000 1 0 2024-09-13T12:10:15.603 INFO:tasks.workunit.client.0.smithi032.stdout:1.15 3 0 0 0 0 1449 0 0 14 30 14 active+clean 2024-09-13T12:09:54.403068+0000 38'44 56:142 [2,0,1] 2 [2,0,1] 2 38'44 2024-09-13T12:09:54.402988+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:17:36.286752+0000 3 0 2024-09-13T12:10:15.603 INFO:tasks.workunit.client.0.smithi032.stdout:1.16 4 0 0 0 0 1932 0 0 16 30 16 active+clean 2024-09-13T12:10:00.648951+0000 38'46 54:144 [1,0,2] 1 [1,0,2] 1 38'46 2024-09-13T12:10:00.648885+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T16:15:31.391301+0000 4 0 2024-09-13T12:10:15.603 INFO:tasks.workunit.client.0.smithi032.stdout:1.17 1 0 0 0 0 483 0 0 13 30 13 active+clean 2024-09-13T12:09:56.659063+0000 32'43 54:138 [1,2,0] 1 [1,2,0] 1 32'43 2024-09-13T12:09:56.659002+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T12:20:54.948230+0000 1 0 2024-09-13T12:10:15.603 INFO:tasks.workunit.client.0.smithi032.stdout:1.18 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:10:01.491543+0000 44'45 54:150 [1,2,0] 1 [1,2,0] 1 32'43 2024-09-13T12:09:47.765518+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T22:32:25.893239+0000 1 0 2024-09-13T12:10:15.603 INFO:tasks.workunit.client.0.smithi032.stdout:1.19 2 0 0 0 0 966 0 0 14 30 14 active+clean 2024-09-13T12:10:07.700975+0000 34'44 54:115 [1,2,0] 1 [1,2,0] 1 34'44 2024-09-13T12:10:07.700914+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T21:09:06.707898+0000 2 0 2024-09-13T12:10:15.603 INFO:tasks.workunit.client.0.smithi032.stdout:1.1a 4 0 0 0 0 1932 0 0 15 30 15 active+clean 2024-09-13T12:10:05.687691+0000 41'45 54:136 [1,2,0] 1 [1,2,0] 1 41'45 2024-09-13T12:10:05.687638+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T12:27:41.381490+0000 4 0 2024-09-13T12:10:15.603 INFO:tasks.workunit.client.0.smithi032.stdout:1.1b 6 0 0 0 0 2086 0 0 18 30 18 active+clean 2024-09-13T12:10:07.059202+0000 51'48 54:157 [1,0,2] 1 [1,0,2] 1 36'45 2024-09-13T12:09:54.677751+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T18:05:45.271168+0000 4 0 2024-09-13T12:10:15.603 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:10:15.603 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:10:15.603 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:10:15.612 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2024-09-13T12:10:15.613 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:10:15.613 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:10:15.613 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:10:15.613 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:10:15.613 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:10:15.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:10:15.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-09-13T12:10:15.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.0 2024-09-13T12:10:16.290 INFO:tasks.workunit.client.0.smithi032.stderr:osd.0 is already down. 2024-09-13T12:10:16.301 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-09-13T12:10:16.301 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.1 2024-09-13T12:10:16.843 INFO:tasks.workunit.client.0.smithi032.stderr:osd.1 is already down. 2024-09-13T12:10:16.855 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-09-13T12:10:16.855 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.2 2024-09-13T12:10:17.370 INFO:tasks.workunit.client.0.smithi032.stderr:osd.2 is already down. 2024-09-13T12:10:17.381 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-09-13T12:10:17.381 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.0 2024-09-13T12:10:17.910 INFO:tasks.workunit.client.0.smithi032.stderr:osd.0 is already out. 2024-09-13T12:10:17.921 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-09-13T12:10:17.921 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.2 2024-09-13T12:10:18.477 INFO:tasks.workunit.client.0.smithi032.stderr:osd.2 is already out. 2024-09-13T12:10:18.488 INFO:tasks.workunit.client.0.smithi032.stdout:bringing up non_divergent 0 2 2024-09-13T12:10:18.489 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'bringing up non_divergent 0 2' 2024-09-13T12:10:18.489 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2024-09-13T12:10:18.903 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:10:18.903 INFO:tasks.workunit.client.0.smithi032.stdout:1.1f 3 0 0 0 0 1449 0 0 15 30 15 stale+active+clean 2024-09-13T12:09:53.714446+0000 38'45 56:158 [1,2,0] 1 [1,2,0] 1 38'45 2024-09-13T12:09:53.714380+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:06:40.197030+0000 3 0 2024-09-13T12:10:18.903 INFO:tasks.workunit.client.0.smithi032.stdout:1.1e 3 0 0 0 0 1449 0 0 15 30 15 stale+active+clean 2024-09-13T12:10:01.695484+0000 44'45 56:139 [1,0,2] 1 [1,0,2] 1 44'45 2024-09-13T12:10:01.695428+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-15T00:02:21.702027+0000 3 0 2024-09-13T12:10:18.903 INFO:tasks.workunit.client.0.smithi032.stdout:1.1d 4 0 0 0 0 1932 0 0 16 30 16 stale+activating 2024-09-13T12:10:10.851143+0000 39'46 56:154 [2,1,0] 2 [2,1,0] 2 34'44 2024-09-13T12:09:49.397412+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 no scrub is scheduled 2 0 2024-09-13T12:10:18.903 INFO:tasks.workunit.client.0.smithi032.stdout:1.b 3 0 0 0 0 1449 0 0 14 30 14 stale+active+clean 2024-09-13T12:10:07.059118+0000 42'44 56:150 [1,0,2] 1 [1,0,2] 1 42'44 2024-09-13T12:09:57.774856+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T16:35:36.378474+0000 3 0 2024-09-13T12:10:18.903 INFO:tasks.workunit.client.0.smithi032.stdout:1.a 3 0 0 0 0 1449 0 0 14 30 14 stale+active+clean 2024-09-13T12:10:06.700195+0000 34'44 56:120 [1,2,0] 1 [1,2,0] 1 34'44 2024-09-13T12:10:06.700135+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T18:03:20.957867+0000 3 0 2024-09-13T12:10:18.903 INFO:tasks.workunit.client.0.smithi032.stdout:1.9 5 0 0 0 0 2086 0 0 16 30 16 stale+active+clean 2024-09-13T12:10:03.691468+0000 51'46 56:144 [1,2,0] 1 [1,2,0] 1 42'45 2024-09-13T12:10:03.691408+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:46:08.598313+0000 5 0 2024-09-13T12:10:18.904 INFO:tasks.workunit.client.0.smithi032.stdout:1.8 3 0 0 0 0 1449 0 0 16 30 16 stale+active+clean 2024-09-13T12:10:01.491569+0000 42'46 56:156 [1,2,0] 1 [1,2,0] 1 42'46 2024-09-13T12:09:58.652148+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T18:38:04.829153+0000 3 0 2024-09-13T12:10:18.904 INFO:tasks.workunit.client.0.smithi032.stdout:1.7 2 0 0 0 0 966 0 0 14 30 14 stale+active+clean 2024-09-13T12:09:55.719638+0000 34'44 56:144 [1,2,0] 1 [1,2,0] 1 34'44 2024-09-13T12:09:55.719570+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T23:56:51.321893+0000 2 0 2024-09-13T12:10:18.904 INFO:tasks.workunit.client.0.smithi032.stdout:1.6 4 0 0 0 0 1932 0 0 16 30 16 stale+active+clean 2024-09-13T12:10:02.648923+0000 41'46 56:158 [1,0,2] 1 [1,0,2] 1 41'46 2024-09-13T12:10:02.648875+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:26:25.904524+0000 4 0 2024-09-13T12:10:18.904 INFO:tasks.workunit.client.0.smithi032.stdout:1.5 6 0 0 0 0 2898 0 0 16 30 16 stale+active+clean 2024-09-13T12:09:52.375479+0000 42'46 56:144 [2,0,1] 2 [2,0,1] 2 38'45 2024-09-13T12:09:52.375427+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T12:21:52.219300+0000 5 0 2024-09-13T12:10:18.904 INFO:tasks.workunit.client.0.smithi032.stdout:1.4 2 0 0 0 0 966 0 0 14 30 14 stale+active+clean 2024-09-13T12:09:46.701281+0000 34'44 56:148 [1,0,2] 1 [1,0,2] 1 34'44 2024-09-13T12:09:46.701189+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T21:51:06.640681+0000 2 0 2024-09-13T12:10:18.904 INFO:tasks.workunit.client.0.smithi032.stdout:1.3 5 0 0 0 0 2415 0 0 15 30 15 stale+active+clean 2024-09-13T12:09:52.263095+0000 44'45 56:145 [1,2,0] 1 [1,2,0] 1 32'43 2024-09-13T12:09:44.638103+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T20:57:24.639697+0000 3 0 2024-09-13T12:10:18.904 INFO:tasks.workunit.client.0.smithi032.stdout:1.2 1 0 0 0 0 483 0 0 14 30 14 stale+active+clean 2024-09-13T12:09:55.946647+0000 39'44 56:137 [0,1,2] 0 [0,1,2] 0 39'44 2024-09-13T12:09:55.946588+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:35:10.767429+0000 1 0 2024-09-13T12:10:18.904 INFO:tasks.workunit.client.0.smithi032.stdout:1.c 3 0 0 0 0 1449 0 0 15 30 15 stale+active+clean 2024-09-13T12:09:51.776961+0000 41'45 56:149 [1,0,2] 1 [1,0,2] 1 34'44 2024-09-13T12:09:51.776896+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T23:00:42.045128+0000 2 0 2024-09-13T12:10:18.904 INFO:tasks.workunit.client.0.smithi032.stdout:1.1 2 0 0 0 0 966 0 0 15 30 15 stale+active+clean 2024-09-13T12:09:48.455915+0000 41'45 56:143 [2,0,1] 2 [2,0,1] 2 32'43 2024-09-13T12:09:48.455862+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T14:27:09.423237+0000 0 0 2024-09-13T12:10:18.904 INFO:tasks.workunit.client.0.smithi032.stdout:1.1c 2 0 0 0 0 966 0 0 14 30 14 stale+active+clean 2024-09-13T12:09:49.691499+0000 34'44 56:148 [1,0,2] 1 [1,0,2] 1 34'44 2024-09-13T12:09:49.691404+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:06:39.817186+0000 2 0 2024-09-13T12:10:18.904 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 3 0 0 0 0 1449 0 0 14 30 14 stale+active+clean 2024-09-13T12:09:43.710429+0000 42'44 56:148 [1,0,2] 1 [1,0,2] 1 32'43 2024-09-13T12:09:43.710370+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:10:53.500857+0000 2 0 2024-09-13T12:10:18.904 INFO:tasks.workunit.client.0.smithi032.stdout:1.d 3 0 0 0 0 1449 0 0 15 30 15 stale+activating 2024-09-13T12:10:10.851135+0000 42'45 56:142 [2,1,0] 2 [2,1,0] 2 32'43 2024-09-13T12:09:50.728150+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 no scrub is scheduled 1 0 2024-09-13T12:10:18.905 INFO:tasks.workunit.client.0.smithi032.stdout:1.e 4 0 0 0 0 1449 0 0 17 30 17 stale+active+clean 2024-09-13T12:09:57.574280+0000 51'47 56:145 [1,0,2] 1 [1,0,2] 1 32'43 2024-09-13T12:09:49.907160+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:55:37.354668+0000 2 0 2024-09-13T12:10:18.905 INFO:tasks.workunit.client.0.smithi032.stdout:1.f 4 0 0 0 0 1932 0 0 14 30 14 stale+active+clean 2024-09-13T12:09:59.616237+0000 38'44 56:146 [1,2,0] 1 [1,2,0] 1 38'44 2024-09-13T12:09:59.616191+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T19:28:06.424316+0000 4 0 2024-09-13T12:10:18.905 INFO:tasks.workunit.client.0.smithi032.stdout:1.10 3 0 0 0 0 1449 0 0 15 30 15 stale+active+clean 2024-09-13T12:09:52.759410+0000 43'45 56:149 [1,0,2] 1 [1,0,2] 1 34'44 2024-09-13T12:09:52.759350+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T18:22:49.943296+0000 2 0 2024-09-13T12:10:18.905 INFO:tasks.workunit.client.0.smithi032.stdout:1.11 2 0 0 0 0 966 0 0 16 30 16 stale+active+clean 2024-09-13T12:09:53.421673+0000 51'46 56:155 [2,0,1] 2 [2,0,1] 2 34'45 2024-09-13T12:09:53.421579+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T12:40:23.459149+0000 3 0 2024-09-13T12:10:18.905 INFO:tasks.workunit.client.0.smithi032.stdout:1.12 1 0 0 0 0 483 0 0 14 30 14 stale+active+clean 2024-09-13T12:09:52.858712+0000 34'44 56:148 [0,1,2] 0 [0,1,2] 0 34'44 2024-09-13T12:09:52.858658+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T17:10:31.364184+0000 1 0 2024-09-13T12:10:18.905 INFO:tasks.workunit.client.0.smithi032.stdout:1.13 3 0 0 0 0 1449 0 0 15 30 15 stale+active+clean 2024-09-13T12:10:04.717979+0000 44'45 56:147 [1,2,0] 1 [1,2,0] 1 44'45 2024-09-13T12:10:04.717933+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T15:08:00.991049+0000 3 0 2024-09-13T12:10:18.905 INFO:tasks.workunit.client.0.smithi032.stdout:1.14 3 0 0 0 0 1449 0 0 15 30 15 stale+active+clean 2024-09-13T12:09:50.728046+0000 44'45 56:149 [1,0,2] 1 [1,0,2] 1 32'43 2024-09-13T12:09:50.727967+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T22:32:42.500192+0000 1 0 2024-09-13T12:10:18.905 INFO:tasks.workunit.client.0.smithi032.stdout:1.15 3 0 0 0 0 1449 0 0 14 30 14 stale+active+clean 2024-09-13T12:09:54.403068+0000 38'44 56:142 [2,0,1] 2 [2,0,1] 2 38'44 2024-09-13T12:09:54.402988+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:17:36.286752+0000 3 0 2024-09-13T12:10:18.905 INFO:tasks.workunit.client.0.smithi032.stdout:1.16 4 0 0 0 0 1932 0 0 16 30 16 stale+active+clean 2024-09-13T12:10:00.648951+0000 38'46 56:149 [1,0,2] 1 [1,0,2] 1 38'46 2024-09-13T12:10:00.648885+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T14:50:07.703237+0000 4 0 2024-09-13T12:10:18.905 INFO:tasks.workunit.client.0.smithi032.stdout:1.17 1 0 0 0 0 483 0 0 13 30 13 stale+active+clean 2024-09-13T12:09:56.659063+0000 32'43 56:143 [1,2,0] 1 [1,2,0] 1 32'43 2024-09-13T12:09:56.659002+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:38:57.651549+0000 1 0 2024-09-13T12:10:18.905 INFO:tasks.workunit.client.0.smithi032.stdout:1.18 3 0 0 0 0 1449 0 0 15 30 15 stale+active+clean 2024-09-13T12:10:01.491543+0000 44'45 56:155 [1,2,0] 1 [1,2,0] 1 32'43 2024-09-13T12:09:47.765518+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T22:45:25.179075+0000 1 0 2024-09-13T12:10:18.905 INFO:tasks.workunit.client.0.smithi032.stdout:1.19 2 0 0 0 0 966 0 0 14 30 14 stale+active+clean 2024-09-13T12:10:07.700975+0000 34'44 56:120 [1,2,0] 1 [1,2,0] 1 34'44 2024-09-13T12:10:07.700914+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T23:56:47.636876+0000 2 0 2024-09-13T12:10:18.905 INFO:tasks.workunit.client.0.smithi032.stdout:1.1a 4 0 0 0 0 1932 0 0 15 30 15 stale+active+clean 2024-09-13T12:10:05.687691+0000 41'45 56:141 [1,2,0] 1 [1,2,0] 1 41'45 2024-09-13T12:10:05.687638+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T13:26:40.583115+0000 4 0 2024-09-13T12:10:18.906 INFO:tasks.workunit.client.0.smithi032.stdout:1.1b 6 0 0 0 0 2086 0 0 18 30 18 stale+active+clean 2024-09-13T12:10:07.059202+0000 51'48 56:162 [1,0,2] 1 [1,0,2] 1 36'45 2024-09-13T12:09:54.677751+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T21:03:55.743684+0000 4 0 2024-09-13T12:10:18.906 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:10:18.906 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:10:18.906 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:10:18.917 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-09-13T12:10:18.917 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 0 2024-09-13T12:10:18.917 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-13T12:10:18.917 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:10:18.918 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T12:10:18.918 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:10:18.918 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/0 2024-09-13T12:10:18.918 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T12:10:18.918 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:10:18.918 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:10:18.918 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:10:18.918 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-13T12:10:18.918 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-13T12:10:18.918 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:10:18.918 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:10:18.918 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T12:10:18.918 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:10:18.918 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:10:18.919 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:10:18.919 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:10:18.919 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:10:18.919 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:10:18.919 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:10:18.919 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:10:18.919 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:10:18.919 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T12:10:18.919 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T12:10:18.919 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:10:18.919 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:10:18.919 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:10:18.919 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:10:18.919 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:10:18.920 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:10:18.920 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/0 2024-09-13T12:10:18.920 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.0 2024-09-13T12:10:18.921 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T12:10:18.921 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:10:18.921 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/0/whoami 2024-09-13T12:10:18.921 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T12:10:18.921 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:10:18.921 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:10:18.921 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:10:18.952 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:10:18.951+0000 7fbd40e2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:10:18.954 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:10:18.953+0000 7fbd40e2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:10:18.955 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:10:18.955+0000 7fbd40e2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:10:19.359 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T12:10:19.359 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:10:19.359 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T12:10:19.359 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:10:19.359 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:10:19.359 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:10:19.360 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:10:19.360 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:10:19.360 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:10:19.360 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:10:19.809 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:10:20.047 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:10:20.046+0000 7fbd40e2c640 -1 Falling back to public interface 2024-09-13T12:10:20.811 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:10:20.811 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:10:20.811 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:10:20.811 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:10:20.812 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:10:20.812 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:10:21.261 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:10:22.262 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:10:22.262 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:10:22.263 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:10:22.263 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:10:22.263 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:10:22.263 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:10:22.681 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:10:23.446 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:10:23.445+0000 7fbd40e2c640 -1 osd.0 50 log_to_monitors true 2024-09-13T12:10:23.683 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:10:23.683 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:10:23.683 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:10:23.684 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:10:23.684 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:10:23.684 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:10:24.154 INFO:tasks.workunit.client.0.smithi032.stdout:osd.0 up out weight 0 up_from 63 up_thru 37 down_at 57 last_clean_interval [5,50) [v2:127.0.0.1:6802/2302613541,v1:127.0.0.1:6803/2302613541] [v2:127.0.0.1:6804/2302613541,v1:127.0.0.1:6805/2302613541] exists,up 10a21bfc-e16b-4a73-8068-b8ae59576a72 2024-09-13T12:10:24.157 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:10:24.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:10:24.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:10:24.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-09-13T12:10:24.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 2 2024-09-13T12:10:24.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-13T12:10:24.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:10:24.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-13T12:10:24.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:10:24.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/2 2024-09-13T12:10:24.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T12:10:24.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:10:24.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:10:24.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:10:24.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-13T12:10:24.159 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-13T12:10:24.159 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:10:24.159 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:10:24.159 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T12:10:24.159 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:10:24.159 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:10:24.159 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:10:24.159 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:10:24.159 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:10:24.159 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:10:24.159 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:10:24.159 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:10:24.159 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:10:24.159 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T12:10:24.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T12:10:24.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:10:24.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:10:24.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:10:24.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:10:24.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:10:24.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:10:24.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/2 2024-09-13T12:10:24.160 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.2 2024-09-13T12:10:24.161 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-13T12:10:24.161 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:10:24.161 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/2/whoami 2024-09-13T12:10:24.161 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-13T12:10:24.161 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:10:24.162 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:10:24.162 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:10:24.193 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:10:24.192+0000 7f205c019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:10:24.194 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:10:24.194+0000 7f205c019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:10:24.195 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:10:24.195+0000 7f205c019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:10:24.587 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-13T12:10:24.588 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:10:24.588 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T12:10:24.588 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:10:24.588 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:10:24.588 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:10:24.588 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:10:24.588 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:10:24.589 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:10:24.589 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:10:25.031 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:10:25.030+0000 7f205c019640 -1 Falling back to public interface 2024-09-13T12:10:25.031 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:10:26.033 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:10:26.033 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:10:26.033 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:10:26.033 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:10:26.034 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:10:26.034 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:10:26.507 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:10:27.509 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:10:27.509 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:10:27.510 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:10:27.510 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:10:27.510 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:10:27.510 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:10:27.920 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:10:28.060 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:10:28.058+0000 7f205c019640 -1 osd.2 51 log_to_monitors true 2024-09-13T12:10:28.922 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:10:28.922 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:10:28.922 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:10:28.922 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:10:28.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:10:28.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:10:29.337 INFO:tasks.workunit.client.0.smithi032.stdout:osd.2 up out weight 0 up_from 65 up_thru 55 down_at 59 last_clean_interval [15,51) [v2:127.0.0.1:6810/73367264,v1:127.0.0.1:6811/73367264] [v2:127.0.0.1:6812/73367264,v1:127.0.0.1:6813/73367264] exists,up 6ac797ab-1924-4ff8-9169-021a47e84482 2024-09-13T12:10:29.337 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:10:29.337 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:10:29.338 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:10:29.338 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-09-13T12:10:29.338 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.0 2024-09-13T12:10:30.068 INFO:tasks.workunit.client.0.smithi032.stderr:osd.0 is already in. 2024-09-13T12:10:30.080 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-09-13T12:10:30.081 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.2 2024-09-13T12:10:31.049 INFO:tasks.workunit.client.0.smithi032.stderr:osd.2 is already in. 2024-09-13T12:10:31.061 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-13T12:10:31.061 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: wait_for_clean 2024-09-13T12:10:31.061 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T12:10:31.061 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T12:10:31.062 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T12:10:31.062 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-13T12:10:31.062 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T12:10:31.062 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T12:10:31.062 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T12:10:31.063 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T12:10:31.063 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T12:10:31.119 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-13T12:10:31.119 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T12:10:31.120 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T12:10:31.120 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T12:10:31.120 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T12:10:31.120 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T12:10:31.646 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T12:10:31.647 INFO:tasks.workunit.client.0.smithi032.stderr:1 2024-09-13T12:10:31.647 INFO:tasks.workunit.client.0.smithi032.stderr:2' 2024-09-13T12:10:31.647 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T12:10:31.647 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:10:31.647 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T12:10:31.832 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=270582939652 2024-09-13T12:10:31.832 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 270582939652 2024-09-13T12:10:31.832 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-270582939652' 2024-09-13T12:10:31.833 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:10:31.833 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T12:10:31.981 INFO:tasks.workunit.client.0.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-13T12:10:31.983 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-13T12:10:31.983 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-13T12:10:31.983 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-13T12:10:31.983 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:10:31.983 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T12:10:32.191 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=279172874243 2024-09-13T12:10:32.191 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 279172874243 2024-09-13T12:10:32.191 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-270582939652 2-279172874243' 2024-09-13T12:10:32.191 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:10:32.191 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-270582939652 2024-09-13T12:10:32.191 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:10:32.192 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T12:10:32.193 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-270582939652 2024-09-13T12:10:32.193 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:10:32.194 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=270582939652 2024-09-13T12:10:32.194 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 270582939652' 2024-09-13T12:10:32.194 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.0 seq 270582939652 2024-09-13T12:10:32.194 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:10:32.703 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 270582939652 -lt 270582939652 2024-09-13T12:10:32.703 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:10:32.704 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-279172874243 2024-09-13T12:10:32.704 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:10:32.705 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T12:10:32.705 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-279172874243 2024-09-13T12:10:32.705 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:10:32.706 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=279172874243 2024-09-13T12:10:32.706 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 279172874243' 2024-09-13T12:10:32.707 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.2 seq 279172874243 2024-09-13T12:10:32.707 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T12:10:33.155 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 279172874243 -lt 279172874243 2024-09-13T12:10:33.155 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T12:10:33.155 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:10:33.155 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:10:33.721 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 32 == 0 2024-09-13T12:10:33.721 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:10:33.721 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:10:33.721 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:10:33.721 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:10:33.721 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:10:33.721 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:10:33.721 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:10:34.125 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:10:34.126 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:10:34.126 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:10:34.126 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:10:34.683 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-13T12:10:34.683 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-13T12:10:34.683 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T12:10:34.683 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-13T12:10:34.683 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:10:34.683 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T12:10:34.784 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:10:34.784 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:10:34.784 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:10:34.784 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:10:34.785 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:10:34.785 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:10:34.785 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:10:34.785 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:10:35.190 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:10:35.191 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:10:35.191 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:10:35.191 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:10:35.748 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-13T12:10:35.748 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:10:35.748 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:10:35.748 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:10:35.748 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:10:35.748 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:10:35.748 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:10:35.748 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:10:35.748 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:10:36.313 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:10:36.313 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:10:36.313 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-13T12:10:36.314 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:10:36.314 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T12:10:36.515 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:10:36.515 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:10:36.515 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:10:36.516 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:10:36.516 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:10:36.516 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:10:36.516 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:10:36.516 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:10:36.918 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:10:36.919 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:10:36.919 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:10:36.919 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:10:37.474 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-13T12:10:37.474 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:10:37.474 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:10:37.474 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:10:37.474 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:10:37.474 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:10:37.474 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:10:37.474 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:10:37.474 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:10:38.033 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:10:38.033 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:10:38.033 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-13T12:10:38.033 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:10:38.033 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-13T12:10:38.434 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:10:38.434 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:10:38.435 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:10:38.435 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:10:38.435 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:10:38.435 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:10:38.435 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:10:38.435 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:10:38.839 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:10:38.840 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:10:38.840 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:10:38.840 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:10:39.402 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-13T12:10:39.402 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:10:39.402 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:10:39.402 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:10:39.403 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:10:39.403 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:10:39.403 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:10:39.403 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:10:39.403 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:10:39.970 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:10:39.970 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:10:39.970 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-13T12:10:39.970 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:10:39.970 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-13T12:10:40.771 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:10:40.771 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:10:40.772 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:10:40.772 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:10:40.772 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:10:40.772 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:10:40.772 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:10:40.772 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:10:41.187 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:10:41.188 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:10:41.188 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:10:41.188 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:10:41.751 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-13T12:10:41.752 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:10:41.752 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:10:41.752 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:10:41.752 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:10:41.752 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:10:41.752 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:10:41.752 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:10:41.752 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:10:42.307 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:10:42.307 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:10:42.308 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-13T12:10:42.308 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:10:42.308 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-13T12:10:43.908 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:10:43.908 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:10:43.909 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:10:43.909 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:10:43.909 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:10:43.909 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:10:43.910 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:10:43.910 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:10:44.326 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:10:44.326 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:10:44.327 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:10:44.327 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:10:44.882 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-13T12:10:44.882 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:10:44.882 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:10:44.883 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:10:44.883 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:10:44.883 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:10:44.883 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:10:44.883 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:10:44.883 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:10:45.449 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:10:45.449 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:10:45.449 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-13T12:10:45.449 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:10:45.449 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-13T12:10:48.649 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:10:48.649 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:10:48.650 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:10:48.650 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:10:48.650 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:10:48.650 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:10:48.650 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:10:48.650 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:10:49.064 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:10:49.064 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:10:49.064 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:10:49.065 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:10:49.617 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-13T12:10:49.617 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:10:49.617 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:10:49.617 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:10:49.618 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:10:49.618 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:10:49.618 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:10:49.618 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:10:49.618 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:10:50.184 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:10:50.184 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:10:50.184 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-13T12:10:50.184 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:10:50.184 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-13T12:10:56.585 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:10:56.585 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:10:56.586 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:10:56.586 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:10:56.586 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:10:56.586 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:10:56.586 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:10:56.586 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:10:57.001 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:10:57.001 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:10:57.002 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:10:57.002 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:10:57.597 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-13T12:10:57.597 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:10:57.597 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:10:57.597 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:10:57.597 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:10:57.597 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:10:57.597 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:10:57.597 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:10:57.598 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:10:58.159 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:10:58.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:10:58.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-13T12:10:58.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:10:58.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-13T12:11:05.461 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:11:05.461 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:11:05.462 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:11:05.462 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:11:05.462 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:11:05.462 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:11:05.462 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:11:05.462 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:11:05.877 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:11:05.878 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:11:05.878 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:11:05.878 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:11:06.437 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-13T12:11:06.437 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:11:06.437 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:11:06.437 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:11:06.437 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:11:06.437 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:11:06.437 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:11:06.437 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:11:06.437 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:11:06.995 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:11:06.995 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:11:06.995 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-13T12:11:06.995 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-13T12:11:07.567 INFO:tasks.workunit.client.0.smithi032.stdout:{ 2024-09-13T12:11:07.567 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_fingerprint": "30b75177-5678-43f5-989c-4de966e73592", 2024-09-13T12:11:07.568 INFO:tasks.workunit.client.0.smithi032.stdout: "version": "19.3.0-4697-gd58f7306", 2024-09-13T12:11:07.568 INFO:tasks.workunit.client.0.smithi032.stdout: "commit": "d58f73069a8157a9ace12a0247ee3e8497707ea9", 2024-09-13T12:11:07.568 INFO:tasks.workunit.client.0.smithi032.stdout: "timestamp": "2024-09-13T12:11:07.566092+0000", 2024-09-13T12:11:07.568 INFO:tasks.workunit.client.0.smithi032.stdout: "tag": "", 2024-09-13T12:11:07.568 INFO:tasks.workunit.client.0.smithi032.stdout: "health": { 2024-09-13T12:11:07.568 INFO:tasks.workunit.client.0.smithi032.stdout: "status": "HEALTH_WARN", 2024-09-13T12:11:07.568 INFO:tasks.workunit.client.0.smithi032.stdout: "checks": { 2024-09-13T12:11:07.568 INFO:tasks.workunit.client.0.smithi032.stdout: "OSDMAP_FLAGS": { 2024-09-13T12:11:07.568 INFO:tasks.workunit.client.0.smithi032.stdout: "severity": "HEALTH_WARN", 2024-09-13T12:11:07.568 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": { 2024-09-13T12:11:07.569 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-13T12:11:07.569 INFO:tasks.workunit.client.0.smithi032.stdout: "count": 17 2024-09-13T12:11:07.569 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.569 INFO:tasks.workunit.client.0.smithi032.stdout: "detail": [], 2024-09-13T12:11:07.569 INFO:tasks.workunit.client.0.smithi032.stdout: "muted": false 2024-09-13T12:11:07.569 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.570 INFO:tasks.workunit.client.0.smithi032.stdout: "OSD_DOWN": { 2024-09-13T12:11:07.570 INFO:tasks.workunit.client.0.smithi032.stdout: "severity": "HEALTH_WARN", 2024-09-13T12:11:07.570 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": { 2024-09-13T12:11:07.570 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "1 osds down", 2024-09-13T12:11:07.570 INFO:tasks.workunit.client.0.smithi032.stdout: "count": 1 2024-09-13T12:11:07.570 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.570 INFO:tasks.workunit.client.0.smithi032.stdout: "detail": [ 2024-09-13T12:11:07.571 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.571 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "osd.1 (root=default,host=smithi032) is down" 2024-09-13T12:11:07.571 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.571 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:07.571 INFO:tasks.workunit.client.0.smithi032.stdout: "muted": false 2024-09-13T12:11:07.571 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.571 INFO:tasks.workunit.client.0.smithi032.stdout: "PG_DEGRADED": { 2024-09-13T12:11:07.571 INFO:tasks.workunit.client.0.smithi032.stdout: "severity": "HEALTH_WARN", 2024-09-13T12:11:07.571 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": { 2024-09-13T12:11:07.571 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "Degraded data redundancy: 103/309 objects degraded (33.333%), 32 pgs degraded", 2024-09-13T12:11:07.572 INFO:tasks.workunit.client.0.smithi032.stdout: "count": 32 2024-09-13T12:11:07.572 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.572 INFO:tasks.workunit.client.0.smithi032.stdout: "detail": [ 2024-09-13T12:11:07.572 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.572 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-09-13T12:11:07.572 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.572 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.572 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [2,0]" 2024-09-13T12:11:07.572 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.572 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.572 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0,2]" 2024-09-13T12:11:07.573 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.573 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.573 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [2,0]" 2024-09-13T12:11:07.573 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.573 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.573 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [0,2]" 2024-09-13T12:11:07.573 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.573 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.573 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [2,0]" 2024-09-13T12:11:07.573 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.573 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.574 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [0,2]" 2024-09-13T12:11:07.574 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.574 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.574 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [2,0]" 2024-09-13T12:11:07.574 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.574 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.574 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [2,0]" 2024-09-13T12:11:07.575 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.575 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.575 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.9 is active+undersized+degraded, acting [2,0]" 2024-09-13T12:11:07.575 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.575 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.576 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.a is active+undersized+degraded, acting [2,0]" 2024-09-13T12:11:07.576 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.576 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.577 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.b is active+undersized+degraded, acting [0,2]" 2024-09-13T12:11:07.577 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.577 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.577 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.c is active+undersized+degraded, acting [0,2]" 2024-09-13T12:11:07.577 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.578 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.578 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.d is active+undersized+degraded, acting [2,0]" 2024-09-13T12:11:07.578 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.579 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.579 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.e is active+undersized+degraded, acting [0,2]" 2024-09-13T12:11:07.579 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.579 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.579 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.f is active+undersized+degraded, acting [2,0]" 2024-09-13T12:11:07.579 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.579 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.579 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [0,2]" 2024-09-13T12:11:07.580 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.580 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.580 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [2,0]" 2024-09-13T12:11:07.580 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.580 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.580 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [0,2]" 2024-09-13T12:11:07.580 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.580 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.580 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [2,0]" 2024-09-13T12:11:07.580 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.580 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.581 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [0,2]" 2024-09-13T12:11:07.581 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.581 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.581 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [2,0]" 2024-09-13T12:11:07.581 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.581 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.582 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.16 is active+undersized+degraded, acting [0,2]" 2024-09-13T12:11:07.582 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.582 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.582 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.17 is active+undersized+degraded, acting [2,0]" 2024-09-13T12:11:07.582 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.582 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.583 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.18 is active+undersized+degraded, acting [2,0]" 2024-09-13T12:11:07.583 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.583 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.583 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.19 is active+undersized+degraded, acting [2,0]" 2024-09-13T12:11:07.583 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.583 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.583 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.1a is active+undersized+degraded, acting [2,0]" 2024-09-13T12:11:07.583 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.583 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.583 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.1b is active+undersized+degraded, acting [0,2]" 2024-09-13T12:11:07.583 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.583 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.584 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [0,2]" 2024-09-13T12:11:07.584 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.584 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.584 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [2,0]" 2024-09-13T12:11:07.584 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.584 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.584 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [0,2]" 2024-09-13T12:11:07.584 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.584 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.584 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [2,0]" 2024-09-13T12:11:07.585 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.585 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:07.585 INFO:tasks.workunit.client.0.smithi032.stdout: "muted": false 2024-09-13T12:11:07.585 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.585 INFO:tasks.workunit.client.0.smithi032.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-13T12:11:07.585 INFO:tasks.workunit.client.0.smithi032.stdout: "severity": "HEALTH_WARN", 2024-09-13T12:11:07.585 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": { 2024-09-13T12:11:07.585 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-13T12:11:07.585 INFO:tasks.workunit.client.0.smithi032.stdout: "count": 1 2024-09-13T12:11:07.585 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.585 INFO:tasks.workunit.client.0.smithi032.stdout: "detail": [ 2024-09-13T12:11:07.585 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.586 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "application not enabled on pool 'test'" 2024-09-13T12:11:07.586 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.586 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.586 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-13T12:11:07.586 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.586 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:07.586 INFO:tasks.workunit.client.0.smithi032.stdout: "muted": false 2024-09-13T12:11:07.586 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.586 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.587 INFO:tasks.workunit.client.0.smithi032.stdout: "mutes": [] 2024-09-13T12:11:07.587 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.587 INFO:tasks.workunit.client.0.smithi032.stdout: "monmap_first_committed": 1, 2024-09-13T12:11:07.587 INFO:tasks.workunit.client.0.smithi032.stdout: "monmap_last_committed": 1, 2024-09-13T12:11:07.587 INFO:tasks.workunit.client.0.smithi032.stdout: "monmap": { 2024-09-13T12:11:07.587 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 1, 2024-09-13T12:11:07.587 INFO:tasks.workunit.client.0.smithi032.stdout: "fsid": "e12dfbc3-9bf5-45db-a6f0-f25db93ee351", 2024-09-13T12:11:07.587 INFO:tasks.workunit.client.0.smithi032.stdout: "modified": "2024-09-13T12:07:34.499882Z", 2024-09-13T12:11:07.587 INFO:tasks.workunit.client.0.smithi032.stdout: "created": "2024-09-13T12:07:34.499882Z", 2024-09-13T12:11:07.587 INFO:tasks.workunit.client.0.smithi032.stdout: "min_mon_release": 19, 2024-09-13T12:11:07.587 INFO:tasks.workunit.client.0.smithi032.stdout: "min_mon_release_name": "squid", 2024-09-13T12:11:07.587 INFO:tasks.workunit.client.0.smithi032.stdout: "election_strategy": 1, 2024-09-13T12:11:07.587 INFO:tasks.workunit.client.0.smithi032.stdout: "disallowed_leaders: ": "", 2024-09-13T12:11:07.587 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_mode": false, 2024-09-13T12:11:07.588 INFO:tasks.workunit.client.0.smithi032.stdout: "tiebreaker_mon": "", 2024-09-13T12:11:07.588 INFO:tasks.workunit.client.0.smithi032.stdout: "removed_ranks: ": "", 2024-09-13T12:11:07.588 INFO:tasks.workunit.client.0.smithi032.stdout: "features": { 2024-09-13T12:11:07.588 INFO:tasks.workunit.client.0.smithi032.stdout: "persistent": [ 2024-09-13T12:11:07.588 INFO:tasks.workunit.client.0.smithi032.stdout: "kraken", 2024-09-13T12:11:07.588 INFO:tasks.workunit.client.0.smithi032.stdout: "luminous", 2024-09-13T12:11:07.588 INFO:tasks.workunit.client.0.smithi032.stdout: "mimic", 2024-09-13T12:11:07.588 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap-prune", 2024-09-13T12:11:07.588 INFO:tasks.workunit.client.0.smithi032.stdout: "nautilus", 2024-09-13T12:11:07.589 INFO:tasks.workunit.client.0.smithi032.stdout: "octopus", 2024-09-13T12:11:07.589 INFO:tasks.workunit.client.0.smithi032.stdout: "pacific", 2024-09-13T12:11:07.589 INFO:tasks.workunit.client.0.smithi032.stdout: "elector-pinging", 2024-09-13T12:11:07.589 INFO:tasks.workunit.client.0.smithi032.stdout: "quincy", 2024-09-13T12:11:07.589 INFO:tasks.workunit.client.0.smithi032.stdout: "reef", 2024-09-13T12:11:07.589 INFO:tasks.workunit.client.0.smithi032.stdout: "squid" 2024-09-13T12:11:07.589 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:07.589 INFO:tasks.workunit.client.0.smithi032.stdout: "optional": [] 2024-09-13T12:11:07.590 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.590 INFO:tasks.workunit.client.0.smithi032.stdout: "mons": [ 2024-09-13T12:11:07.590 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.590 INFO:tasks.workunit.client.0.smithi032.stdout: "rank": 0, 2024-09-13T12:11:07.590 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "a", 2024-09-13T12:11:07.590 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addrs": { 2024-09-13T12:11:07.591 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:11:07.591 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.591 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:11:07.591 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:7115", 2024-09-13T12:11:07.591 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 0 2024-09-13T12:11:07.591 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.591 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:07.592 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.592 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:7115/0", 2024-09-13T12:11:07.592 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-13T12:11:07.592 INFO:tasks.workunit.client.0.smithi032.stdout: "priority": 0, 2024-09-13T12:11:07.592 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 0, 2024-09-13T12:11:07.593 INFO:tasks.workunit.client.0.smithi032.stdout: "crush_location": "{}" 2024-09-13T12:11:07.593 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.593 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:07.593 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.594 INFO:tasks.workunit.client.0.smithi032.stdout: "quorum": [ 2024-09-13T12:11:07.594 INFO:tasks.workunit.client.0.smithi032.stdout: 0 2024-09-13T12:11:07.594 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:07.594 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap": { 2024-09-13T12:11:07.594 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 73, 2024-09-13T12:11:07.595 INFO:tasks.workunit.client.0.smithi032.stdout: "fsid": "e12dfbc3-9bf5-45db-a6f0-f25db93ee351", 2024-09-13T12:11:07.595 INFO:tasks.workunit.client.0.smithi032.stdout: "created": "2024-09-13T12:07:35.402712+0000", 2024-09-13T12:11:07.595 INFO:tasks.workunit.client.0.smithi032.stdout: "modified": "2024-09-13T12:10:31.792953+0000", 2024-09-13T12:11:07.595 INFO:tasks.workunit.client.0.smithi032.stdout: "last_up_change": "2024-09-13T12:10:28.520213+0000", 2024-09-13T12:11:07.595 INFO:tasks.workunit.client.0.smithi032.stdout: "last_in_change": "2024-09-13T12:10:30.500123+0000", 2024-09-13T12:11:07.595 INFO:tasks.workunit.client.0.smithi032.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T12:11:07.596 INFO:tasks.workunit.client.0.smithi032.stdout: "flags_num": 5800384, 2024-09-13T12:11:07.596 INFO:tasks.workunit.client.0.smithi032.stdout: "flags_set": [ 2024-09-13T12:11:07.596 INFO:tasks.workunit.client.0.smithi032.stdout: "nodown", 2024-09-13T12:11:07.596 INFO:tasks.workunit.client.0.smithi032.stdout: "noin", 2024-09-13T12:11:07.596 INFO:tasks.workunit.client.0.smithi032.stdout: "noout", 2024-09-13T12:11:07.597 INFO:tasks.workunit.client.0.smithi032.stdout: "pglog_hardlimit", 2024-09-13T12:11:07.597 INFO:tasks.workunit.client.0.smithi032.stdout: "purged_snapdirs", 2024-09-13T12:11:07.597 INFO:tasks.workunit.client.0.smithi032.stdout: "recovery_deletes", 2024-09-13T12:11:07.597 INFO:tasks.workunit.client.0.smithi032.stdout: "sortbitwise" 2024-09-13T12:11:07.597 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:07.597 INFO:tasks.workunit.client.0.smithi032.stdout: "crush_version": 9, 2024-09-13T12:11:07.598 INFO:tasks.workunit.client.0.smithi032.stdout: "full_ratio": 0.99000000953674316, 2024-09-13T12:11:07.598 INFO:tasks.workunit.client.0.smithi032.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-13T12:11:07.598 INFO:tasks.workunit.client.0.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T12:11:07.598 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_snapshot": "", 2024-09-13T12:11:07.598 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_max": 1, 2024-09-13T12:11:07.599 INFO:tasks.workunit.client.0.smithi032.stdout: "max_osd": 3, 2024-09-13T12:11:07.599 INFO:tasks.workunit.client.0.smithi032.stdout: "require_min_compat_client": "luminous", 2024-09-13T12:11:07.599 INFO:tasks.workunit.client.0.smithi032.stdout: "min_compat_client": "jewel", 2024-09-13T12:11:07.599 INFO:tasks.workunit.client.0.smithi032.stdout: "require_osd_release": "squid", 2024-09-13T12:11:07.599 INFO:tasks.workunit.client.0.smithi032.stdout: "allow_crimson": false, 2024-09-13T12:11:07.599 INFO:tasks.workunit.client.0.smithi032.stdout: "pools": [ 2024-09-13T12:11:07.600 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.600 INFO:tasks.workunit.client.0.smithi032.stdout: "pool": 1, 2024-09-13T12:11:07.600 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_name": "test", 2024-09-13T12:11:07.600 INFO:tasks.workunit.client.0.smithi032.stdout: "create_time": "2024-09-13T12:09:23.852748+0000", 2024-09-13T12:11:07.600 INFO:tasks.workunit.client.0.smithi032.stdout: "flags": 1, 2024-09-13T12:11:07.601 INFO:tasks.workunit.client.0.smithi032.stdout: "flags_names": "hashpspool", 2024-09-13T12:11:07.601 INFO:tasks.workunit.client.0.smithi032.stdout: "type": 1, 2024-09-13T12:11:07.601 INFO:tasks.workunit.client.0.smithi032.stdout: "size": 3, 2024-09-13T12:11:07.601 INFO:tasks.workunit.client.0.smithi032.stdout: "min_size": 2, 2024-09-13T12:11:07.601 INFO:tasks.workunit.client.0.smithi032.stdout: "crush_rule": 0, 2024-09-13T12:11:07.601 INFO:tasks.workunit.client.0.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-13T12:11:07.602 INFO:tasks.workunit.client.0.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-13T12:11:07.602 INFO:tasks.workunit.client.0.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T12:11:07.602 INFO:tasks.workunit.client.0.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T12:11:07.603 INFO:tasks.workunit.client.0.smithi032.stdout: "is_stretch_pool": false, 2024-09-13T12:11:07.603 INFO:tasks.workunit.client.0.smithi032.stdout: "object_hash": 2, 2024-09-13T12:11:07.603 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_autoscale_mode": "on", 2024-09-13T12:11:07.603 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_num": 32, 2024-09-13T12:11:07.603 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_placement_num": 14, 2024-09-13T12:11:07.604 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_placement_num_target": 32, 2024-09-13T12:11:07.604 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_num_target": 32, 2024-09-13T12:11:07.604 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_num_pending": 32, 2024-09-13T12:11:07.604 INFO:tasks.workunit.client.0.smithi032.stdout: "last_pg_merge_meta": { 2024-09-13T12:11:07.604 INFO:tasks.workunit.client.0.smithi032.stdout: "source_pgid": "0.0", 2024-09-13T12:11:07.604 INFO:tasks.workunit.client.0.smithi032.stdout: "ready_epoch": 0, 2024-09-13T12:11:07.605 INFO:tasks.workunit.client.0.smithi032.stdout: "last_epoch_started": 0, 2024-09-13T12:11:07.605 INFO:tasks.workunit.client.0.smithi032.stdout: "last_epoch_clean": 0, 2024-09-13T12:11:07.605 INFO:tasks.workunit.client.0.smithi032.stdout: "source_version": "0'0", 2024-09-13T12:11:07.605 INFO:tasks.workunit.client.0.smithi032.stdout: "target_version": "0'0" 2024-09-13T12:11:07.605 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.606 INFO:tasks.workunit.client.0.smithi032.stdout: "last_change": "55", 2024-09-13T12:11:07.606 INFO:tasks.workunit.client.0.smithi032.stdout: "last_force_op_resend": "0", 2024-09-13T12:11:07.606 INFO:tasks.workunit.client.0.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T12:11:07.606 INFO:tasks.workunit.client.0.smithi032.stdout: "last_force_op_resend_preluminous": "33", 2024-09-13T12:11:07.606 INFO:tasks.workunit.client.0.smithi032.stdout: "auid": 0, 2024-09-13T12:11:07.607 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_mode": "selfmanaged", 2024-09-13T12:11:07.607 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_seq": 0, 2024-09-13T12:11:07.607 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_epoch": 0, 2024-09-13T12:11:07.607 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_snaps": [], 2024-09-13T12:11:07.607 INFO:tasks.workunit.client.0.smithi032.stdout: "removed_snaps": "[]", 2024-09-13T12:11:07.607 INFO:tasks.workunit.client.0.smithi032.stdout: "quota_max_bytes": 0, 2024-09-13T12:11:07.608 INFO:tasks.workunit.client.0.smithi032.stdout: "quota_max_objects": 0, 2024-09-13T12:11:07.608 INFO:tasks.workunit.client.0.smithi032.stdout: "tiers": [], 2024-09-13T12:11:07.608 INFO:tasks.workunit.client.0.smithi032.stdout: "tier_of": -1, 2024-09-13T12:11:07.608 INFO:tasks.workunit.client.0.smithi032.stdout: "read_tier": -1, 2024-09-13T12:11:07.608 INFO:tasks.workunit.client.0.smithi032.stdout: "write_tier": -1, 2024-09-13T12:11:07.608 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_mode": "none", 2024-09-13T12:11:07.608 INFO:tasks.workunit.client.0.smithi032.stdout: "target_max_bytes": 0, 2024-09-13T12:11:07.608 INFO:tasks.workunit.client.0.smithi032.stdout: "target_max_objects": 0, 2024-09-13T12:11:07.608 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T12:11:07.608 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T12:11:07.608 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T12:11:07.608 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-13T12:11:07.608 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-13T12:11:07.609 INFO:tasks.workunit.client.0.smithi032.stdout: "erasure_code_profile": "", 2024-09-13T12:11:07.609 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_params": { 2024-09-13T12:11:07.609 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "none" 2024-09-13T12:11:07.609 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.609 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_period": 0, 2024-09-13T12:11:07.609 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_count": 0, 2024-09-13T12:11:07.609 INFO:tasks.workunit.client.0.smithi032.stdout: "use_gmt_hitset": true, 2024-09-13T12:11:07.609 INFO:tasks.workunit.client.0.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-13T12:11:07.609 INFO:tasks.workunit.client.0.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-13T12:11:07.609 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T12:11:07.609 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-13T12:11:07.609 INFO:tasks.workunit.client.0.smithi032.stdout: "grade_table": [], 2024-09-13T12:11:07.609 INFO:tasks.workunit.client.0.smithi032.stdout: "stripe_width": 0, 2024-09-13T12:11:07.610 INFO:tasks.workunit.client.0.smithi032.stdout: "expected_num_objects": 0, 2024-09-13T12:11:07.610 INFO:tasks.workunit.client.0.smithi032.stdout: "fast_read": false, 2024-09-13T12:11:07.610 INFO:tasks.workunit.client.0.smithi032.stdout: "options": {}, 2024-09-13T12:11:07.610 INFO:tasks.workunit.client.0.smithi032.stdout: "application_metadata": {}, 2024-09-13T12:11:07.610 INFO:tasks.workunit.client.0.smithi032.stdout: "read_balance": { 2024-09-13T12:11:07.610 INFO:tasks.workunit.client.0.smithi032.stdout: "score_type": "Fair distribution", 2024-09-13T12:11:07.610 INFO:tasks.workunit.client.0.smithi032.stdout: "score_acting": 1.690000057220459, 2024-09-13T12:11:07.610 INFO:tasks.workunit.client.0.smithi032.stdout: "score_stable": 1.690000057220459, 2024-09-13T12:11:07.610 INFO:tasks.workunit.client.0.smithi032.stdout: "optimal_score": 0.67000001668930054, 2024-09-13T12:11:07.611 INFO:tasks.workunit.client.0.smithi032.stdout: "raw_score_acting": 1.1299999952316284, 2024-09-13T12:11:07.611 INFO:tasks.workunit.client.0.smithi032.stdout: "raw_score_stable": 1.1299999952316284, 2024-09-13T12:11:07.611 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_affinity_weighted": 1, 2024-09-13T12:11:07.611 INFO:tasks.workunit.client.0.smithi032.stdout: "average_primary_affinity": 1, 2024-09-13T12:11:07.611 INFO:tasks.workunit.client.0.smithi032.stdout: "average_primary_affinity_weighted": 1 2024-09-13T12:11:07.611 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.611 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.611 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:07.611 INFO:tasks.workunit.client.0.smithi032.stdout: "osds": [ 2024-09-13T12:11:07.611 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.611 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 0, 2024-09-13T12:11:07.611 INFO:tasks.workunit.client.0.smithi032.stdout: "uuid": "10a21bfc-e16b-4a73-8068-b8ae59576a72", 2024-09-13T12:11:07.611 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 1, 2024-09-13T12:11:07.611 INFO:tasks.workunit.client.0.smithi032.stdout: "in": 1, 2024-09-13T12:11:07.612 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 1, 2024-09-13T12:11:07.612 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_affinity": 1, 2024-09-13T12:11:07.612 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_begin": 5, 2024-09-13T12:11:07.612 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_end": 50, 2024-09-13T12:11:07.612 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 63, 2024-09-13T12:11:07.612 INFO:tasks.workunit.client.0.smithi032.stdout: "up_thru": 69, 2024-09-13T12:11:07.612 INFO:tasks.workunit.client.0.smithi032.stdout: "down_at": 57, 2024-09-13T12:11:07.612 INFO:tasks.workunit.client.0.smithi032.stdout: "lost_at": 0, 2024-09-13T12:11:07.612 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addrs": { 2024-09-13T12:11:07.612 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:11:07.612 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.612 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:11:07.612 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6802", 2024-09-13T12:11:07.613 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 2302613541 2024-09-13T12:11:07.613 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.613 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.613 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:11:07.613 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6803", 2024-09-13T12:11:07.613 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 2302613541 2024-09-13T12:11:07.613 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.613 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:07.613 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.613 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addrs": { 2024-09-13T12:11:07.613 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:11:07.613 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.613 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:11:07.614 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6804", 2024-09-13T12:11:07.614 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 2302613541 2024-09-13T12:11:07.614 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.614 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.614 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:11:07.614 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6805", 2024-09-13T12:11:07.614 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 2302613541 2024-09-13T12:11:07.614 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.614 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:07.614 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.614 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-13T12:11:07.614 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:11:07.614 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.614 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:11:07.615 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6808", 2024-09-13T12:11:07.615 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 2302613541 2024-09-13T12:11:07.615 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.615 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.615 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:11:07.615 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6809", 2024-09-13T12:11:07.615 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 2302613541 2024-09-13T12:11:07.615 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.615 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:07.615 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.615 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-13T12:11:07.616 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:11:07.616 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.616 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:11:07.616 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6806", 2024-09-13T12:11:07.616 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 2302613541 2024-09-13T12:11:07.616 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.616 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.616 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:11:07.617 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6807", 2024-09-13T12:11:07.617 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 2302613541 2024-09-13T12:11:07.617 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.617 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:07.617 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.617 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addr": "127.0.0.1:6803/2302613541", 2024-09-13T12:11:07.617 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addr": "127.0.0.1:6805/2302613541", 2024-09-13T12:11:07.618 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addr": "127.0.0.1:6809/2302613541", 2024-09-13T12:11:07.618 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addr": "127.0.0.1:6807/2302613541", 2024-09-13T12:11:07.618 INFO:tasks.workunit.client.0.smithi032.stdout: "state": [ 2024-09-13T12:11:07.618 INFO:tasks.workunit.client.0.smithi032.stdout: "exists", 2024-09-13T12:11:07.618 INFO:tasks.workunit.client.0.smithi032.stdout: "up" 2024-09-13T12:11:07.618 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:07.618 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.618 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.618 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 1, 2024-09-13T12:11:07.618 INFO:tasks.workunit.client.0.smithi032.stdout: "uuid": "f377d2cc-d182-4459-bd79-b23c9379a371", 2024-09-13T12:11:07.618 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 0, 2024-09-13T12:11:07.618 INFO:tasks.workunit.client.0.smithi032.stdout: "in": 1, 2024-09-13T12:11:07.618 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 1, 2024-09-13T12:11:07.618 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_affinity": 1, 2024-09-13T12:11:07.619 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_begin": 0, 2024-09-13T12:11:07.619 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_end": 0, 2024-09-13T12:11:07.619 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 10, 2024-09-13T12:11:07.619 INFO:tasks.workunit.client.0.smithi032.stdout: "up_thru": 52, 2024-09-13T12:11:07.619 INFO:tasks.workunit.client.0.smithi032.stdout: "down_at": 58, 2024-09-13T12:11:07.619 INFO:tasks.workunit.client.0.smithi032.stdout: "lost_at": 0, 2024-09-13T12:11:07.619 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addrs": { 2024-09-13T12:11:07.619 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:11:07.619 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.619 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:11:07.619 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6810", 2024-09-13T12:11:07.619 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1424248451 2024-09-13T12:11:07.619 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.619 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.620 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:11:07.620 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6811", 2024-09-13T12:11:07.620 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1424248451 2024-09-13T12:11:07.620 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.620 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:07.620 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.620 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addrs": { 2024-09-13T12:11:07.620 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:11:07.620 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.620 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:11:07.620 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6812", 2024-09-13T12:11:07.621 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1424248451 2024-09-13T12:11:07.621 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.621 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.621 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:11:07.621 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6813", 2024-09-13T12:11:07.622 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1424248451 2024-09-13T12:11:07.622 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.622 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:07.622 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.622 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-13T12:11:07.622 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:11:07.622 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.622 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:11:07.622 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6816", 2024-09-13T12:11:07.623 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1424248451 2024-09-13T12:11:07.623 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.623 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.623 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:11:07.623 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6817", 2024-09-13T12:11:07.623 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1424248451 2024-09-13T12:11:07.623 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.623 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:07.623 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.623 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-13T12:11:07.624 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:11:07.624 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.624 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:11:07.624 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6814", 2024-09-13T12:11:07.624 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1424248451 2024-09-13T12:11:07.624 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.624 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.624 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:11:07.625 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6815", 2024-09-13T12:11:07.625 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1424248451 2024-09-13T12:11:07.625 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.625 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:07.625 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.625 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addr": "127.0.0.1:6811/1424248451", 2024-09-13T12:11:07.625 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addr": "127.0.0.1:6813/1424248451", 2024-09-13T12:11:07.625 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1424248451", 2024-09-13T12:11:07.625 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1424248451", 2024-09-13T12:11:07.625 INFO:tasks.workunit.client.0.smithi032.stdout: "state": [ 2024-09-13T12:11:07.625 INFO:tasks.workunit.client.0.smithi032.stdout: "exists" 2024-09-13T12:11:07.625 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:07.625 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.626 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.626 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2, 2024-09-13T12:11:07.626 INFO:tasks.workunit.client.0.smithi032.stdout: "uuid": "6ac797ab-1924-4ff8-9169-021a47e84482", 2024-09-13T12:11:07.626 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 1, 2024-09-13T12:11:07.626 INFO:tasks.workunit.client.0.smithi032.stdout: "in": 1, 2024-09-13T12:11:07.626 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 1, 2024-09-13T12:11:07.626 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_affinity": 1, 2024-09-13T12:11:07.628 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_begin": 15, 2024-09-13T12:11:07.628 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_end": 51, 2024-09-13T12:11:07.628 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 65, 2024-09-13T12:11:07.628 INFO:tasks.workunit.client.0.smithi032.stdout: "up_thru": 71, 2024-09-13T12:11:07.628 INFO:tasks.workunit.client.0.smithi032.stdout: "down_at": 59, 2024-09-13T12:11:07.628 INFO:tasks.workunit.client.0.smithi032.stdout: "lost_at": 0, 2024-09-13T12:11:07.628 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addrs": { 2024-09-13T12:11:07.628 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:11:07.628 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.628 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:11:07.629 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6810", 2024-09-13T12:11:07.629 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 73367264 2024-09-13T12:11:07.629 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.629 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.629 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:11:07.629 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6811", 2024-09-13T12:11:07.629 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 73367264 2024-09-13T12:11:07.629 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.629 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:07.629 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.629 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addrs": { 2024-09-13T12:11:07.629 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:11:07.629 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.630 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:11:07.630 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6812", 2024-09-13T12:11:07.630 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 73367264 2024-09-13T12:11:07.630 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.630 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.630 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:11:07.630 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6813", 2024-09-13T12:11:07.630 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 73367264 2024-09-13T12:11:07.630 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.630 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:07.630 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.630 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-13T12:11:07.630 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:11:07.631 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.631 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:11:07.631 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6816", 2024-09-13T12:11:07.631 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 73367264 2024-09-13T12:11:07.631 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.631 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.631 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:11:07.631 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6817", 2024-09-13T12:11:07.631 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 73367264 2024-09-13T12:11:07.631 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.631 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:07.631 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.632 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-13T12:11:07.632 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:11:07.632 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.632 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:11:07.632 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6814", 2024-09-13T12:11:07.632 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 73367264 2024-09-13T12:11:07.632 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.632 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.632 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:11:07.632 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6815", 2024-09-13T12:11:07.632 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 73367264 2024-09-13T12:11:07.633 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.633 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:07.633 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.633 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addr": "127.0.0.1:6811/73367264", 2024-09-13T12:11:07.633 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addr": "127.0.0.1:6813/73367264", 2024-09-13T12:11:07.633 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addr": "127.0.0.1:6817/73367264", 2024-09-13T12:11:07.633 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addr": "127.0.0.1:6815/73367264", 2024-09-13T12:11:07.633 INFO:tasks.workunit.client.0.smithi032.stdout: "state": [ 2024-09-13T12:11:07.633 INFO:tasks.workunit.client.0.smithi032.stdout: "exists", 2024-09-13T12:11:07.633 INFO:tasks.workunit.client.0.smithi032.stdout: "up" 2024-09-13T12:11:07.633 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:07.633 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.633 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:07.634 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_xinfo": [ 2024-09-13T12:11:07.634 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.634 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 0, 2024-09-13T12:11:07.634 INFO:tasks.workunit.client.0.smithi032.stdout: "down_stamp": "2024-09-13T12:10:16.065537+0000", 2024-09-13T12:11:07.634 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_probability": 0, 2024-09-13T12:11:07.634 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_interval": 0, 2024-09-13T12:11:07.634 INFO:tasks.workunit.client.0.smithi032.stdout: "features": 4540719139924082687, 2024-09-13T12:11:07.634 INFO:tasks.workunit.client.0.smithi032.stdout: "old_weight": 0, 2024-09-13T12:11:07.634 INFO:tasks.workunit.client.0.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-13T12:08:09.703357+0000", 2024-09-13T12:11:07.634 INFO:tasks.workunit.client.0.smithi032.stdout: "dead_epoch": 0 2024-09-13T12:11:07.634 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.634 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.634 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 1, 2024-09-13T12:11:07.635 INFO:tasks.workunit.client.0.smithi032.stdout: "down_stamp": "2024-09-13T12:10:16.671856+0000", 2024-09-13T12:11:07.635 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_probability": 0, 2024-09-13T12:11:07.635 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_interval": 0, 2024-09-13T12:11:07.635 INFO:tasks.workunit.client.0.smithi032.stdout: "features": 4540719139924082687, 2024-09-13T12:11:07.635 INFO:tasks.workunit.client.0.smithi032.stdout: "old_weight": 0, 2024-09-13T12:11:07.635 INFO:tasks.workunit.client.0.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-13T12:08:44.286568+0000", 2024-09-13T12:11:07.635 INFO:tasks.workunit.client.0.smithi032.stdout: "dead_epoch": 0 2024-09-13T12:11:07.635 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.635 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.635 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2, 2024-09-13T12:11:07.635 INFO:tasks.workunit.client.0.smithi032.stdout: "down_stamp": "2024-09-13T12:10:17.200480+0000", 2024-09-13T12:11:07.635 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_probability": 0, 2024-09-13T12:11:07.635 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_interval": 0, 2024-09-13T12:11:07.635 INFO:tasks.workunit.client.0.smithi032.stdout: "features": 4540719139924082687, 2024-09-13T12:11:07.636 INFO:tasks.workunit.client.0.smithi032.stdout: "old_weight": 0, 2024-09-13T12:11:07.636 INFO:tasks.workunit.client.0.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-13T12:09:20.289467+0000", 2024-09-13T12:11:07.636 INFO:tasks.workunit.client.0.smithi032.stdout: "dead_epoch": 0 2024-09-13T12:11:07.636 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.636 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:07.636 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_upmap": [], 2024-09-13T12:11:07.636 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_upmap_items": [], 2024-09-13T12:11:07.636 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_upmap_primaries": [], 2024-09-13T12:11:07.636 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_temp": [], 2024-09-13T12:11:07.636 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_temp": [], 2024-09-13T12:11:07.636 INFO:tasks.workunit.client.0.smithi032.stdout: "blocklist": {}, 2024-09-13T12:11:07.636 INFO:tasks.workunit.client.0.smithi032.stdout: "range_blocklist": {}, 2024-09-13T12:11:07.636 INFO:tasks.workunit.client.0.smithi032.stdout: "erasure_code_profiles": { 2024-09-13T12:11:07.637 INFO:tasks.workunit.client.0.smithi032.stdout: "default": { 2024-09-13T12:11:07.637 INFO:tasks.workunit.client.0.smithi032.stdout: "k": "2", 2024-09-13T12:11:07.637 INFO:tasks.workunit.client.0.smithi032.stdout: "m": "2", 2024-09-13T12:11:07.637 INFO:tasks.workunit.client.0.smithi032.stdout: "plugin": "jerasure", 2024-09-13T12:11:07.637 INFO:tasks.workunit.client.0.smithi032.stdout: "technique": "reed_sol_van" 2024-09-13T12:11:07.637 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.637 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.637 INFO:tasks.workunit.client.0.smithi032.stdout: "removed_snaps_queue": [], 2024-09-13T12:11:07.637 INFO:tasks.workunit.client.0.smithi032.stdout: "new_removed_snaps": [], 2024-09-13T12:11:07.637 INFO:tasks.workunit.client.0.smithi032.stdout: "new_purged_snaps": [], 2024-09-13T12:11:07.637 INFO:tasks.workunit.client.0.smithi032.stdout: "crush_node_flags": {}, 2024-09-13T12:11:07.638 INFO:tasks.workunit.client.0.smithi032.stdout: "device_class_flags": {}, 2024-09-13T12:11:07.638 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_mode": { 2024-09-13T12:11:07.638 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_mode_enabled": false, 2024-09-13T12:11:07.638 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_bucket_count": 0, 2024-09-13T12:11:07.638 INFO:tasks.workunit.client.0.smithi032.stdout: "degraded_stretch_mode": 0, 2024-09-13T12:11:07.638 INFO:tasks.workunit.client.0.smithi032.stdout: "recovering_stretch_mode": 0, 2024-09-13T12:11:07.638 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_mode_bucket": 0 2024-09-13T12:11:07.638 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.638 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.638 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_metadata": [ 2024-09-13T12:11:07.638 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.638 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:11:07.638 INFO:tasks.workunit.client.0.smithi032.stdout: "arch": "x86_64", 2024-09-13T12:11:07.639 INFO:tasks.workunit.client.0.smithi032.stdout: "back_addr": "[v2:127.0.0.1:6804/2302613541,v1:127.0.0.1:6805/2302613541]", 2024-09-13T12:11:07.639 INFO:tasks.workunit.client.0.smithi032.stdout: "back_iface": "", 2024-09-13T12:11:07.639 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs": "1", 2024-09-13T12:11:07.639 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_db": "0", 2024-09-13T12:11:07.639 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_wal": "0", 2024-09-13T12:11:07.639 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_single_shared_device": "1", 2024-09-13T12:11:07.639 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_allocation_from_file": "0", 2024-09-13T12:11:07.639 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_access_mode": "file", 2024-09-13T12:11:07.639 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_block_size": "4096", 2024-09-13T12:11:07.639 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_devices": "sda", 2024-09-13T12:11:07.639 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-13T12:11:07.639 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-13T12:11:07.639 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-13T12:11:07.639 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_rotational": "1", 2024-09-13T12:11:07.639 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_size": "107374182400", 2024-09-13T12:11:07.640 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_support_discard": "1", 2024-09-13T12:11:07.640 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_type": "hdd", 2024-09-13T12:11:07.640 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_min_alloc_size": "4096", 2024-09-13T12:11:07.640 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_release": "squid", 2024-09-13T12:11:07.640 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:11:07.640 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_short": "19.3.0-4697-gd58f7306", 2024-09-13T12:11:07.640 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_when_created": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:11:07.640 INFO:tasks.workunit.client.0.smithi032.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-13T12:11:07.640 INFO:tasks.workunit.client.0.smithi032.stdout: "created_at": "2024-09-13T12:07:41.934841Z", 2024-09-13T12:11:07.640 INFO:tasks.workunit.client.0.smithi032.stdout: "default_device_class": "hdd", 2024-09-13T12:11:07.640 INFO:tasks.workunit.client.0.smithi032.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4J2LL", 2024-09-13T12:11:07.640 INFO:tasks.workunit.client.0.smithi032.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-13T12:11:07.640 INFO:tasks.workunit.client.0.smithi032.stdout: "devices": "sda", 2024-09-13T12:11:07.641 INFO:tasks.workunit.client.0.smithi032.stdout: "distro": "centos", 2024-09-13T12:11:07.641 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_description": "CentOS Stream 9", 2024-09-13T12:11:07.641 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_version": "9", 2024-09-13T12:11:07.641 INFO:tasks.workunit.client.0.smithi032.stdout: "front_addr": "[v2:127.0.0.1:6802/2302613541,v1:127.0.0.1:6803/2302613541]", 2024-09-13T12:11:07.641 INFO:tasks.workunit.client.0.smithi032.stdout: "front_iface": "", 2024-09-13T12:11:07.641 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/2302613541,v1:127.0.0.1:6809/2302613541]", 2024-09-13T12:11:07.641 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/2302613541,v1:127.0.0.1:6807/2302613541]", 2024-09-13T12:11:07.642 INFO:tasks.workunit.client.0.smithi032.stdout: "hostname": "smithi032", 2024-09-13T12:11:07.642 INFO:tasks.workunit.client.0.smithi032.stdout: "journal_rotational": "1", 2024-09-13T12:11:07.642 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-13T12:11:07.642 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-13T12:11:07.642 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_swap_kb": "0", 2024-09-13T12:11:07.642 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_total_kb": "32493484", 2024-09-13T12:11:07.642 INFO:tasks.workunit.client.0.smithi032.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-13T12:11:07.642 INFO:tasks.workunit.client.0.smithi032.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-13T12:11:07.642 INFO:tasks.workunit.client.0.smithi032.stdout: "os": "Linux", 2024-09-13T12:11:07.642 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_data": "td/divergent-priors/0", 2024-09-13T12:11:07.642 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_objectstore": "bluestore", 2024-09-13T12:11:07.642 INFO:tasks.workunit.client.0.smithi032.stdout: "osdspec_affinity": "", 2024-09-13T12:11:07.642 INFO:tasks.workunit.client.0.smithi032.stdout: "rotational": "1" 2024-09-13T12:11:07.643 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.643 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.643 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 1, 2024-09-13T12:11:07.643 INFO:tasks.workunit.client.0.smithi032.stdout: "arch": "x86_64", 2024-09-13T12:11:07.643 INFO:tasks.workunit.client.0.smithi032.stdout: "back_addr": "[v2:127.0.0.1:6812/1424248451,v1:127.0.0.1:6813/1424248451]", 2024-09-13T12:11:07.643 INFO:tasks.workunit.client.0.smithi032.stdout: "back_iface": "", 2024-09-13T12:11:07.643 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs": "1", 2024-09-13T12:11:07.643 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_db": "0", 2024-09-13T12:11:07.643 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_wal": "0", 2024-09-13T12:11:07.643 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_single_shared_device": "1", 2024-09-13T12:11:07.643 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_allocation_from_file": "0", 2024-09-13T12:11:07.643 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_access_mode": "file", 2024-09-13T12:11:07.643 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_block_size": "4096", 2024-09-13T12:11:07.644 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_devices": "sda", 2024-09-13T12:11:07.644 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-13T12:11:07.644 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-13T12:11:07.644 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-13T12:11:07.644 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_rotational": "1", 2024-09-13T12:11:07.644 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_size": "107374182400", 2024-09-13T12:11:07.644 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_support_discard": "1", 2024-09-13T12:11:07.644 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_type": "hdd", 2024-09-13T12:11:07.644 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_min_alloc_size": "4096", 2024-09-13T12:11:07.644 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_release": "squid", 2024-09-13T12:11:07.644 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:11:07.644 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_short": "19.3.0-4697-gd58f7306", 2024-09-13T12:11:07.645 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_when_created": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:11:07.645 INFO:tasks.workunit.client.0.smithi032.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-13T12:11:07.645 INFO:tasks.workunit.client.0.smithi032.stdout: "created_at": "2024-09-13T12:08:16.197013Z", 2024-09-13T12:11:07.645 INFO:tasks.workunit.client.0.smithi032.stdout: "default_device_class": "hdd", 2024-09-13T12:11:07.645 INFO:tasks.workunit.client.0.smithi032.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4J2LL", 2024-09-13T12:11:07.645 INFO:tasks.workunit.client.0.smithi032.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-13T12:11:07.645 INFO:tasks.workunit.client.0.smithi032.stdout: "devices": "sda", 2024-09-13T12:11:07.645 INFO:tasks.workunit.client.0.smithi032.stdout: "distro": "centos", 2024-09-13T12:11:07.645 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_description": "CentOS Stream 9", 2024-09-13T12:11:07.645 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_version": "9", 2024-09-13T12:11:07.645 INFO:tasks.workunit.client.0.smithi032.stdout: "front_addr": "[v2:127.0.0.1:6810/1424248451,v1:127.0.0.1:6811/1424248451]", 2024-09-13T12:11:07.646 INFO:tasks.workunit.client.0.smithi032.stdout: "front_iface": "", 2024-09-13T12:11:07.646 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1424248451,v1:127.0.0.1:6817/1424248451]", 2024-09-13T12:11:07.646 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1424248451,v1:127.0.0.1:6815/1424248451]", 2024-09-13T12:11:07.646 INFO:tasks.workunit.client.0.smithi032.stdout: "hostname": "smithi032", 2024-09-13T12:11:07.646 INFO:tasks.workunit.client.0.smithi032.stdout: "journal_rotational": "1", 2024-09-13T12:11:07.646 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-13T12:11:07.646 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-13T12:11:07.646 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_swap_kb": "0", 2024-09-13T12:11:07.646 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_total_kb": "32493484", 2024-09-13T12:11:07.646 INFO:tasks.workunit.client.0.smithi032.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-13T12:11:07.646 INFO:tasks.workunit.client.0.smithi032.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-13T12:11:07.646 INFO:tasks.workunit.client.0.smithi032.stdout: "os": "Linux", 2024-09-13T12:11:07.646 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_data": "td/divergent-priors/1", 2024-09-13T12:11:07.647 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_objectstore": "bluestore", 2024-09-13T12:11:07.647 INFO:tasks.workunit.client.0.smithi032.stdout: "osdspec_affinity": "", 2024-09-13T12:11:07.647 INFO:tasks.workunit.client.0.smithi032.stdout: "rotational": "1" 2024-09-13T12:11:07.647 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.647 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.647 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:11:07.647 INFO:tasks.workunit.client.0.smithi032.stdout: "arch": "x86_64", 2024-09-13T12:11:07.647 INFO:tasks.workunit.client.0.smithi032.stdout: "back_addr": "[v2:127.0.0.1:6812/73367264,v1:127.0.0.1:6813/73367264]", 2024-09-13T12:11:07.647 INFO:tasks.workunit.client.0.smithi032.stdout: "back_iface": "", 2024-09-13T12:11:07.647 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs": "1", 2024-09-13T12:11:07.647 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_db": "0", 2024-09-13T12:11:07.647 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_wal": "0", 2024-09-13T12:11:07.647 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_single_shared_device": "1", 2024-09-13T12:11:07.647 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_allocation_from_file": "0", 2024-09-13T12:11:07.648 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_access_mode": "file", 2024-09-13T12:11:07.648 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_block_size": "4096", 2024-09-13T12:11:07.648 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_devices": "sda", 2024-09-13T12:11:07.648 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-13T12:11:07.648 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-13T12:11:07.648 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-13T12:11:07.648 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_rotational": "1", 2024-09-13T12:11:07.648 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_size": "107374182400", 2024-09-13T12:11:07.648 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_support_discard": "1", 2024-09-13T12:11:07.648 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_type": "hdd", 2024-09-13T12:11:07.648 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_min_alloc_size": "4096", 2024-09-13T12:11:07.648 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_release": "squid", 2024-09-13T12:11:07.649 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:11:07.649 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_short": "19.3.0-4697-gd58f7306", 2024-09-13T12:11:07.649 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_when_created": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:11:07.649 INFO:tasks.workunit.client.0.smithi032.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-13T12:11:07.649 INFO:tasks.workunit.client.0.smithi032.stdout: "created_at": "2024-09-13T12:08:52.300025Z", 2024-09-13T12:11:07.649 INFO:tasks.workunit.client.0.smithi032.stdout: "default_device_class": "hdd", 2024-09-13T12:11:07.649 INFO:tasks.workunit.client.0.smithi032.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4J2LL", 2024-09-13T12:11:07.649 INFO:tasks.workunit.client.0.smithi032.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-13T12:11:07.649 INFO:tasks.workunit.client.0.smithi032.stdout: "devices": "sda", 2024-09-13T12:11:07.649 INFO:tasks.workunit.client.0.smithi032.stdout: "distro": "centos", 2024-09-13T12:11:07.650 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_description": "CentOS Stream 9", 2024-09-13T12:11:07.650 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_version": "9", 2024-09-13T12:11:07.650 INFO:tasks.workunit.client.0.smithi032.stdout: "front_addr": "[v2:127.0.0.1:6810/73367264,v1:127.0.0.1:6811/73367264]", 2024-09-13T12:11:07.650 INFO:tasks.workunit.client.0.smithi032.stdout: "front_iface": "", 2024-09-13T12:11:07.650 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/73367264,v1:127.0.0.1:6817/73367264]", 2024-09-13T12:11:07.650 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/73367264,v1:127.0.0.1:6815/73367264]", 2024-09-13T12:11:07.650 INFO:tasks.workunit.client.0.smithi032.stdout: "hostname": "smithi032", 2024-09-13T12:11:07.650 INFO:tasks.workunit.client.0.smithi032.stdout: "journal_rotational": "1", 2024-09-13T12:11:07.650 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-13T12:11:07.650 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-13T12:11:07.650 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_swap_kb": "0", 2024-09-13T12:11:07.650 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_total_kb": "32493484", 2024-09-13T12:11:07.650 INFO:tasks.workunit.client.0.smithi032.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-13T12:11:07.651 INFO:tasks.workunit.client.0.smithi032.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-13T12:11:07.651 INFO:tasks.workunit.client.0.smithi032.stdout: "os": "Linux", 2024-09-13T12:11:07.651 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_data": "td/divergent-priors/2", 2024-09-13T12:11:07.651 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_objectstore": "bluestore", 2024-09-13T12:11:07.651 INFO:tasks.workunit.client.0.smithi032.stdout: "osdspec_affinity": "", 2024-09-13T12:11:07.651 INFO:tasks.workunit.client.0.smithi032.stdout: "rotational": "1" 2024-09-13T12:11:07.651 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.651 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:07.651 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap_clean_epochs": { 2024-09-13T12:11:07.651 INFO:tasks.workunit.client.0.smithi032.stdout: "min_last_epoch_clean": 0, 2024-09-13T12:11:07.651 INFO:tasks.workunit.client.0.smithi032.stdout: "last_epoch_clean": { 2024-09-13T12:11:07.651 INFO:tasks.workunit.client.0.smithi032.stdout: "per_pool": [ 2024-09-13T12:11:07.652 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.652 INFO:tasks.workunit.client.0.smithi032.stdout: "poolid": 1, 2024-09-13T12:11:07.652 INFO:tasks.workunit.client.0.smithi032.stdout: "floor": 36 2024-09-13T12:11:07.652 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.652 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:07.652 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.652 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_epochs": [ 2024-09-13T12:11:07.652 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.652 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:11:07.652 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 63 2024-09-13T12:11:07.653 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.653 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.653 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:11:07.653 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 65 2024-09-13T12:11:07.653 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.653 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:07.653 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.653 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap_first_committed": 1, 2024-09-13T12:11:07.653 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap_last_committed": 73, 2024-09-13T12:11:07.653 INFO:tasks.workunit.client.0.smithi032.stdout: "crushmap": { 2024-09-13T12:11:07.653 INFO:tasks.workunit.client.0.smithi032.stdout: "devices": [ 2024-09-13T12:11:07.654 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.654 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:11:07.654 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "osd.0", 2024-09-13T12:11:07.654 INFO:tasks.workunit.client.0.smithi032.stdout: "class": "hdd" 2024-09-13T12:11:07.654 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.654 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.654 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 1, 2024-09-13T12:11:07.654 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "osd.1", 2024-09-13T12:11:07.654 INFO:tasks.workunit.client.0.smithi032.stdout: "class": "hdd" 2024-09-13T12:11:07.654 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.655 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.655 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:11:07.655 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "osd.2", 2024-09-13T12:11:07.655 INFO:tasks.workunit.client.0.smithi032.stdout: "class": "hdd" 2024-09-13T12:11:07.655 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.655 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:07.655 INFO:tasks.workunit.client.0.smithi032.stdout: "types": [ 2024-09-13T12:11:07.655 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.655 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 0, 2024-09-13T12:11:07.655 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "osd" 2024-09-13T12:11:07.655 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.655 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.655 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 1, 2024-09-13T12:11:07.655 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "host" 2024-09-13T12:11:07.656 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.656 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.656 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 2, 2024-09-13T12:11:07.656 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "chassis" 2024-09-13T12:11:07.656 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.656 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.656 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 3, 2024-09-13T12:11:07.656 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "rack" 2024-09-13T12:11:07.656 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.656 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.656 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 4, 2024-09-13T12:11:07.656 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "row" 2024-09-13T12:11:07.656 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.656 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.657 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 5, 2024-09-13T12:11:07.657 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "pdu" 2024-09-13T12:11:07.657 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.657 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.657 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 6, 2024-09-13T12:11:07.657 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "pod" 2024-09-13T12:11:07.657 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.657 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.657 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 7, 2024-09-13T12:11:07.657 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "room" 2024-09-13T12:11:07.657 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.658 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.658 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 8, 2024-09-13T12:11:07.658 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "datacenter" 2024-09-13T12:11:07.658 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.658 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.658 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 9, 2024-09-13T12:11:07.658 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "zone" 2024-09-13T12:11:07.658 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.658 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.658 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 10, 2024-09-13T12:11:07.658 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "region" 2024-09-13T12:11:07.659 INFO:tasks.workunit.client.0.smithi032.stderr:report 998167944 2024-09-13T12:11:07.659 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.659 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.659 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 11, 2024-09-13T12:11:07.659 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "root" 2024-09-13T12:11:07.659 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.659 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:07.660 INFO:tasks.workunit.client.0.smithi032.stdout: "buckets": [ 2024-09-13T12:11:07.660 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.660 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -1, 2024-09-13T12:11:07.660 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "default", 2024-09-13T12:11:07.660 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 11, 2024-09-13T12:11:07.660 INFO:tasks.workunit.client.0.smithi032.stdout: "type_name": "root", 2024-09-13T12:11:07.660 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:11:07.660 INFO:tasks.workunit.client.0.smithi032.stdout: "alg": "straw2", 2024-09-13T12:11:07.660 INFO:tasks.workunit.client.0.smithi032.stdout: "hash": "rjenkins1", 2024-09-13T12:11:07.660 INFO:tasks.workunit.client.0.smithi032.stdout: "items": [ 2024-09-13T12:11:07.660 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.660 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -3, 2024-09-13T12:11:07.661 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:11:07.661 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 0 2024-09-13T12:11:07.661 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.661 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:07.661 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.661 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.661 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -2, 2024-09-13T12:11:07.661 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "default~hdd", 2024-09-13T12:11:07.661 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 11, 2024-09-13T12:11:07.661 INFO:tasks.workunit.client.0.smithi032.stdout: "type_name": "root", 2024-09-13T12:11:07.661 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:11:07.661 INFO:tasks.workunit.client.0.smithi032.stdout: "alg": "straw2", 2024-09-13T12:11:07.661 INFO:tasks.workunit.client.0.smithi032.stdout: "hash": "rjenkins1", 2024-09-13T12:11:07.661 INFO:tasks.workunit.client.0.smithi032.stdout: "items": [ 2024-09-13T12:11:07.662 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.662 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -4, 2024-09-13T12:11:07.662 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:11:07.662 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 0 2024-09-13T12:11:07.662 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.662 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:07.662 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.662 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.662 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -3, 2024-09-13T12:11:07.662 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "smithi032", 2024-09-13T12:11:07.663 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 1, 2024-09-13T12:11:07.663 INFO:tasks.workunit.client.0.smithi032.stdout: "type_name": "host", 2024-09-13T12:11:07.663 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:11:07.663 INFO:tasks.workunit.client.0.smithi032.stdout: "alg": "straw2", 2024-09-13T12:11:07.663 INFO:tasks.workunit.client.0.smithi032.stdout: "hash": "rjenkins1", 2024-09-13T12:11:07.663 INFO:tasks.workunit.client.0.smithi032.stdout: "items": [ 2024-09-13T12:11:07.663 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.663 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:11:07.663 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:11:07.663 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 0 2024-09-13T12:11:07.663 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.663 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.663 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 1, 2024-09-13T12:11:07.664 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:11:07.664 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 1 2024-09-13T12:11:07.664 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.664 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.664 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:11:07.664 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:11:07.664 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 2 2024-09-13T12:11:07.664 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.664 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:07.665 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.665 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.665 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -4, 2024-09-13T12:11:07.665 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "smithi032~hdd", 2024-09-13T12:11:07.665 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 1, 2024-09-13T12:11:07.665 INFO:tasks.workunit.client.0.smithi032.stdout: "type_name": "host", 2024-09-13T12:11:07.665 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:11:07.665 INFO:tasks.workunit.client.0.smithi032.stdout: "alg": "straw2", 2024-09-13T12:11:07.665 INFO:tasks.workunit.client.0.smithi032.stdout: "hash": "rjenkins1", 2024-09-13T12:11:07.666 INFO:tasks.workunit.client.0.smithi032.stdout: "items": [ 2024-09-13T12:11:07.666 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.666 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:11:07.666 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:11:07.666 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 0 2024-09-13T12:11:07.666 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.666 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.666 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 1, 2024-09-13T12:11:07.666 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:11:07.666 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 1 2024-09-13T12:11:07.666 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.666 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.666 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:11:07.666 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:11:07.667 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 2 2024-09-13T12:11:07.667 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.667 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:07.667 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.667 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:07.667 INFO:tasks.workunit.client.0.smithi032.stdout: "rules": [ 2024-09-13T12:11:07.667 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.667 INFO:tasks.workunit.client.0.smithi032.stdout: "rule_id": 0, 2024-09-13T12:11:07.667 INFO:tasks.workunit.client.0.smithi032.stdout: "rule_name": "replicated_rule", 2024-09-13T12:11:07.667 INFO:tasks.workunit.client.0.smithi032.stdout: "type": 1, 2024-09-13T12:11:07.667 INFO:tasks.workunit.client.0.smithi032.stdout: "steps": [ 2024-09-13T12:11:07.667 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.667 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "take", 2024-09-13T12:11:07.667 INFO:tasks.workunit.client.0.smithi032.stdout: "item": -1, 2024-09-13T12:11:07.668 INFO:tasks.workunit.client.0.smithi032.stdout: "item_name": "default" 2024-09-13T12:11:07.668 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.668 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.668 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "choose_firstn", 2024-09-13T12:11:07.668 INFO:tasks.workunit.client.0.smithi032.stdout: "num": 0, 2024-09-13T12:11:07.668 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "osd" 2024-09-13T12:11:07.668 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.668 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.668 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "emit" 2024-09-13T12:11:07.668 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.669 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:07.669 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.669 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:07.669 INFO:tasks.workunit.client.0.smithi032.stdout: "tunables": { 2024-09-13T12:11:07.669 INFO:tasks.workunit.client.0.smithi032.stdout: "choose_local_tries": 0, 2024-09-13T12:11:07.669 INFO:tasks.workunit.client.0.smithi032.stdout: "choose_local_fallback_tries": 0, 2024-09-13T12:11:07.669 INFO:tasks.workunit.client.0.smithi032.stdout: "choose_total_tries": 50, 2024-09-13T12:11:07.669 INFO:tasks.workunit.client.0.smithi032.stdout: "chooseleaf_descend_once": 1, 2024-09-13T12:11:07.669 INFO:tasks.workunit.client.0.smithi032.stdout: "chooseleaf_vary_r": 1, 2024-09-13T12:11:07.669 INFO:tasks.workunit.client.0.smithi032.stdout: "chooseleaf_stable": 1, 2024-09-13T12:11:07.669 INFO:tasks.workunit.client.0.smithi032.stdout: "msr_descents": 100, 2024-09-13T12:11:07.669 INFO:tasks.workunit.client.0.smithi032.stdout: "msr_collision_tries": 100, 2024-09-13T12:11:07.670 INFO:tasks.workunit.client.0.smithi032.stdout: "straw_calc_version": 1, 2024-09-13T12:11:07.670 INFO:tasks.workunit.client.0.smithi032.stdout: "allowed_bucket_algs": 54, 2024-09-13T12:11:07.670 INFO:tasks.workunit.client.0.smithi032.stdout: "profile": "jewel", 2024-09-13T12:11:07.670 INFO:tasks.workunit.client.0.smithi032.stdout: "optimal_tunables": 1, 2024-09-13T12:11:07.670 INFO:tasks.workunit.client.0.smithi032.stdout: "legacy_tunables": 0, 2024-09-13T12:11:07.670 INFO:tasks.workunit.client.0.smithi032.stdout: "minimum_required_version": "jewel", 2024-09-13T12:11:07.671 INFO:tasks.workunit.client.0.smithi032.stdout: "require_feature_tunables": 1, 2024-09-13T12:11:07.671 INFO:tasks.workunit.client.0.smithi032.stdout: "require_feature_tunables2": 1, 2024-09-13T12:11:07.671 INFO:tasks.workunit.client.0.smithi032.stdout: "has_v2_rules": 0, 2024-09-13T12:11:07.671 INFO:tasks.workunit.client.0.smithi032.stdout: "require_feature_tunables3": 1, 2024-09-13T12:11:07.671 INFO:tasks.workunit.client.0.smithi032.stdout: "has_v3_rules": 0, 2024-09-13T12:11:07.671 INFO:tasks.workunit.client.0.smithi032.stdout: "has_v4_buckets": 1, 2024-09-13T12:11:07.671 INFO:tasks.workunit.client.0.smithi032.stdout: "require_feature_tunables5": 1, 2024-09-13T12:11:07.671 INFO:tasks.workunit.client.0.smithi032.stdout: "has_v5_rules": 0, 2024-09-13T12:11:07.671 INFO:tasks.workunit.client.0.smithi032.stdout: "has_msr_rules": 0 2024-09-13T12:11:07.671 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.671 INFO:tasks.workunit.client.0.smithi032.stdout: "choose_args": {} 2024-09-13T12:11:07.671 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.672 INFO:tasks.workunit.client.0.smithi032.stdout: "fsmap": { 2024-09-13T12:11:07.672 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 1, 2024-09-13T12:11:07.672 INFO:tasks.workunit.client.0.smithi032.stdout: "btime": "2024-09-13T12:07:35:378184+0000", 2024-09-13T12:11:07.672 INFO:tasks.workunit.client.0.smithi032.stdout: "default_fscid": -1, 2024-09-13T12:11:07.672 INFO:tasks.workunit.client.0.smithi032.stdout: "compat": { 2024-09-13T12:11:07.672 INFO:tasks.workunit.client.0.smithi032.stdout: "compat": {}, 2024-09-13T12:11:07.672 INFO:tasks.workunit.client.0.smithi032.stdout: "ro_compat": {}, 2024-09-13T12:11:07.672 INFO:tasks.workunit.client.0.smithi032.stdout: "incompat": { 2024-09-13T12:11:07.672 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_1": "base v0.20", 2024-09-13T12:11:07.672 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_2": "client writeable ranges", 2024-09-13T12:11:07.672 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_3": "default file layouts on dirs", 2024-09-13T12:11:07.673 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_4": "dir inode in separate object", 2024-09-13T12:11:07.673 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_5": "mds uses versioned encoding", 2024-09-13T12:11:07.673 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-13T12:11:07.673 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_8": "no anchor table", 2024-09-13T12:11:07.673 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_9": "file layout v2", 2024-09-13T12:11:07.673 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_10": "snaprealm v2", 2024-09-13T12:11:07.673 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_11": "minor log segments", 2024-09-13T12:11:07.673 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_12": "quiesce subvolumes" 2024-09-13T12:11:07.673 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.673 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.673 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_flags": { 2024-09-13T12:11:07.674 INFO:tasks.workunit.client.0.smithi032.stdout: "enable_multiple": true, 2024-09-13T12:11:07.674 INFO:tasks.workunit.client.0.smithi032.stdout: "ever_enabled_multiple": true 2024-09-13T12:11:07.674 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.674 INFO:tasks.workunit.client.0.smithi032.stdout: "standbys": [], 2024-09-13T12:11:07.674 INFO:tasks.workunit.client.0.smithi032.stdout: "filesystems": [] 2024-09-13T12:11:07.674 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.674 INFO:tasks.workunit.client.0.smithi032.stdout: "mdsmap_first_committed": 1, 2024-09-13T12:11:07.674 INFO:tasks.workunit.client.0.smithi032.stdout: "mdsmap_last_committed": 1, 2024-09-13T12:11:07.674 INFO:tasks.workunit.client.0.smithi032.stdout: "auth": { 2024-09-13T12:11:07.674 INFO:tasks.workunit.client.0.smithi032.stdout: "first_committed": 1, 2024-09-13T12:11:07.674 INFO:tasks.workunit.client.0.smithi032.stdout: "last_committed": 5, 2024-09-13T12:11:07.674 INFO:tasks.workunit.client.0.smithi032.stdout: "num_secrets": 3 2024-09-13T12:11:07.674 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.674 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg": 32, 2024-09-13T12:11:07.675 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg_active": 32, 2024-09-13T12:11:07.675 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg_unknown": 0, 2024-09-13T12:11:07.675 INFO:tasks.workunit.client.0.smithi032.stdout: "num_osd": 3, 2024-09-13T12:11:07.675 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_sum": { 2024-09-13T12:11:07.675 INFO:tasks.workunit.client.0.smithi032.stdout: "stat_sum": { 2024-09-13T12:11:07.675 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes": 48843, 2024-09-13T12:11:07.675 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects": 103, 2024-09-13T12:11:07.675 INFO:tasks.workunit.client.0.smithi032.stdout: "num_object_clones": 0, 2024-09-13T12:11:07.675 INFO:tasks.workunit.client.0.smithi032.stdout: "num_object_copies": 309, 2024-09-13T12:11:07.675 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_missing_on_primary": 0, 2024-09-13T12:11:07.675 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_missing": 0, 2024-09-13T12:11:07.675 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_degraded": 103, 2024-09-13T12:11:07.675 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_misplaced": 0, 2024-09-13T12:11:07.675 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_unfound": 0, 2024-09-13T12:11:07.676 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_dirty": 103, 2024-09-13T12:11:07.676 INFO:tasks.workunit.client.0.smithi032.stdout: "num_whiteouts": 0, 2024-09-13T12:11:07.676 INFO:tasks.workunit.client.0.smithi032.stdout: "num_read": 0, 2024-09-13T12:11:07.676 INFO:tasks.workunit.client.0.smithi032.stdout: "num_read_kb": 0, 2024-09-13T12:11:07.676 INFO:tasks.workunit.client.0.smithi032.stdout: "num_write": 57, 2024-09-13T12:11:07.676 INFO:tasks.workunit.client.0.smithi032.stdout: "num_write_kb": 57, 2024-09-13T12:11:07.676 INFO:tasks.workunit.client.0.smithi032.stdout: "num_scrub_errors": 0, 2024-09-13T12:11:07.676 INFO:tasks.workunit.client.0.smithi032.stdout: "num_shallow_scrub_errors": 0, 2024-09-13T12:11:07.676 INFO:tasks.workunit.client.0.smithi032.stdout: "num_deep_scrub_errors": 0, 2024-09-13T12:11:07.676 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_recovered": 0, 2024-09-13T12:11:07.676 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes_recovered": 0, 2024-09-13T12:11:07.676 INFO:tasks.workunit.client.0.smithi032.stdout: "num_keys_recovered": 0, 2024-09-13T12:11:07.676 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_omap": 0, 2024-09-13T12:11:07.677 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_hit_set_archive": 0, 2024-09-13T12:11:07.677 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes_hit_set_archive": 0, 2024-09-13T12:11:07.677 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush": 0, 2024-09-13T12:11:07.677 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_kb": 0, 2024-09-13T12:11:07.677 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict": 0, 2024-09-13T12:11:07.677 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_kb": 0, 2024-09-13T12:11:07.677 INFO:tasks.workunit.client.0.smithi032.stdout: "num_promote": 0, 2024-09-13T12:11:07.677 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_mode_high": 0, 2024-09-13T12:11:07.677 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_mode_low": 0, 2024-09-13T12:11:07.678 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_mode_some": 0, 2024-09-13T12:11:07.678 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_mode_full": 0, 2024-09-13T12:11:07.678 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_pinned": 0, 2024-09-13T12:11:07.678 INFO:tasks.workunit.client.0.smithi032.stdout: "num_legacy_snapsets": 0, 2024-09-13T12:11:07.678 INFO:tasks.workunit.client.0.smithi032.stdout: "num_large_omap_objects": 0, 2024-09-13T12:11:07.678 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_manifest": 0, 2024-09-13T12:11:07.678 INFO:tasks.workunit.client.0.smithi032.stdout: "num_omap_bytes": 0, 2024-09-13T12:11:07.678 INFO:tasks.workunit.client.0.smithi032.stdout: "num_omap_keys": 0, 2024-09-13T12:11:07.678 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_repaired": 0 2024-09-13T12:11:07.678 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.678 INFO:tasks.workunit.client.0.smithi032.stdout: "store_stats": { 2024-09-13T12:11:07.678 INFO:tasks.workunit.client.0.smithi032.stdout: "total": 0, 2024-09-13T12:11:07.678 INFO:tasks.workunit.client.0.smithi032.stdout: "available": 0, 2024-09-13T12:11:07.678 INFO:tasks.workunit.client.0.smithi032.stdout: "internally_reserved": 0, 2024-09-13T12:11:07.679 INFO:tasks.workunit.client.0.smithi032.stdout: "allocated": 0, 2024-09-13T12:11:07.679 INFO:tasks.workunit.client.0.smithi032.stdout: "data_stored": 0, 2024-09-13T12:11:07.679 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed": 0, 2024-09-13T12:11:07.679 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_allocated": 0, 2024-09-13T12:11:07.679 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_original": 0, 2024-09-13T12:11:07.679 INFO:tasks.workunit.client.0.smithi032.stdout: "omap_allocated": 0, 2024-09-13T12:11:07.679 INFO:tasks.workunit.client.0.smithi032.stdout: "internal_metadata": 0 2024-09-13T12:11:07.679 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.679 INFO:tasks.workunit.client.0.smithi032.stdout: "log_size": 473, 2024-09-13T12:11:07.680 INFO:tasks.workunit.client.0.smithi032.stdout: "ondisk_log_size": 473, 2024-09-13T12:11:07.680 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 64, 2024-09-13T12:11:07.680 INFO:tasks.workunit.client.0.smithi032.stdout: "acting": 64, 2024-09-13T12:11:07.680 INFO:tasks.workunit.client.0.smithi032.stdout: "num_store_stats": 0 2024-09-13T12:11:07.680 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.680 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_sum": { 2024-09-13T12:11:07.680 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 0, 2024-09-13T12:11:07.680 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 0, 2024-09-13T12:11:07.680 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pgs": 96, 2024-09-13T12:11:07.680 INFO:tasks.workunit.client.0.smithi032.stdout: "num_osds": 3, 2024-09-13T12:11:07.680 INFO:tasks.workunit.client.0.smithi032.stdout: "num_per_pool_osds": 3, 2024-09-13T12:11:07.680 INFO:tasks.workunit.client.0.smithi032.stdout: "num_per_pool_omap_osds": 3, 2024-09-13T12:11:07.681 INFO:tasks.workunit.client.0.smithi032.stdout: "kb": 314572800, 2024-09-13T12:11:07.681 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used": 84380, 2024-09-13T12:11:07.681 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_data": 2832, 2024-09-13T12:11:07.681 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_omap": 10, 2024-09-13T12:11:07.681 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_meta": 81461, 2024-09-13T12:11:07.681 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_avail": 314488420, 2024-09-13T12:11:07.681 INFO:tasks.workunit.client.0.smithi032.stdout: "statfs": { 2024-09-13T12:11:07.681 INFO:tasks.workunit.client.0.smithi032.stdout: "total": 322122547200, 2024-09-13T12:11:07.681 INFO:tasks.workunit.client.0.smithi032.stdout: "available": 322036142080, 2024-09-13T12:11:07.681 INFO:tasks.workunit.client.0.smithi032.stdout: "internally_reserved": 0, 2024-09-13T12:11:07.681 INFO:tasks.workunit.client.0.smithi032.stdout: "allocated": 2899968, 2024-09-13T12:11:07.681 INFO:tasks.workunit.client.0.smithi032.stdout: "data_stored": 689175, 2024-09-13T12:11:07.682 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed": 0, 2024-09-13T12:11:07.682 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_allocated": 0, 2024-09-13T12:11:07.682 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_original": 0, 2024-09-13T12:11:07.682 INFO:tasks.workunit.client.0.smithi032.stdout: "omap_allocated": 10396, 2024-09-13T12:11:07.682 INFO:tasks.workunit.client.0.smithi032.stdout: "internal_metadata": 83416932 2024-09-13T12:11:07.682 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.682 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_peers": [], 2024-09-13T12:11:07.682 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_trim_queue_len": 0, 2024-09-13T12:11:07.682 INFO:tasks.workunit.client.0.smithi032.stdout: "num_snap_trimming": 0, 2024-09-13T12:11:07.683 INFO:tasks.workunit.client.0.smithi032.stdout: "num_shards_repaired": 0, 2024-09-13T12:11:07.683 INFO:tasks.workunit.client.0.smithi032.stdout: "op_queue_age_hist": { 2024-09-13T12:11:07.683 INFO:tasks.workunit.client.0.smithi032.stdout: "histogram": [], 2024-09-13T12:11:07.683 INFO:tasks.workunit.client.0.smithi032.stdout: "upper_bound": 1 2024-09-13T12:11:07.683 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.683 INFO:tasks.workunit.client.0.smithi032.stdout: "perf_stat": { 2024-09-13T12:11:07.683 INFO:tasks.workunit.client.0.smithi032.stdout: "commit_latency_ms": 85, 2024-09-13T12:11:07.683 INFO:tasks.workunit.client.0.smithi032.stdout: "apply_latency_ms": 85, 2024-09-13T12:11:07.683 INFO:tasks.workunit.client.0.smithi032.stdout: "commit_latency_ns": 85000000, 2024-09-13T12:11:07.683 INFO:tasks.workunit.client.0.smithi032.stdout: "apply_latency_ns": 85000000 2024-09-13T12:11:07.683 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.683 INFO:tasks.workunit.client.0.smithi032.stdout: "alerts": [], 2024-09-13T12:11:07.683 INFO:tasks.workunit.client.0.smithi032.stdout: "network_ping_times": [] 2024-09-13T12:11:07.683 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.684 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_sum_by_class": { 2024-09-13T12:11:07.684 INFO:tasks.workunit.client.0.smithi032.stdout: "hdd": { 2024-09-13T12:11:07.684 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 0, 2024-09-13T12:11:07.684 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 0, 2024-09-13T12:11:07.684 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pgs": 96, 2024-09-13T12:11:07.684 INFO:tasks.workunit.client.0.smithi032.stdout: "num_osds": 3, 2024-09-13T12:11:07.684 INFO:tasks.workunit.client.0.smithi032.stdout: "num_per_pool_osds": 3, 2024-09-13T12:11:07.684 INFO:tasks.workunit.client.0.smithi032.stdout: "num_per_pool_omap_osds": 3, 2024-09-13T12:11:07.684 INFO:tasks.workunit.client.0.smithi032.stdout: "kb": 314572800, 2024-09-13T12:11:07.684 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used": 84380, 2024-09-13T12:11:07.684 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_data": 2832, 2024-09-13T12:11:07.684 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_omap": 10, 2024-09-13T12:11:07.685 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_meta": 81461, 2024-09-13T12:11:07.685 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_avail": 314488420, 2024-09-13T12:11:07.685 INFO:tasks.workunit.client.0.smithi032.stdout: "statfs": { 2024-09-13T12:11:07.685 INFO:tasks.workunit.client.0.smithi032.stdout: "total": 322122547200, 2024-09-13T12:11:07.685 INFO:tasks.workunit.client.0.smithi032.stdout: "available": 322036142080, 2024-09-13T12:11:07.685 INFO:tasks.workunit.client.0.smithi032.stdout: "internally_reserved": 0, 2024-09-13T12:11:07.685 INFO:tasks.workunit.client.0.smithi032.stdout: "allocated": 2899968, 2024-09-13T12:11:07.685 INFO:tasks.workunit.client.0.smithi032.stdout: "data_stored": 689175, 2024-09-13T12:11:07.685 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed": 0, 2024-09-13T12:11:07.685 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_allocated": 0, 2024-09-13T12:11:07.685 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_original": 0, 2024-09-13T12:11:07.685 INFO:tasks.workunit.client.0.smithi032.stdout: "omap_allocated": 10396, 2024-09-13T12:11:07.685 INFO:tasks.workunit.client.0.smithi032.stdout: "internal_metadata": 83416932 2024-09-13T12:11:07.685 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.686 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_peers": [], 2024-09-13T12:11:07.686 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_trim_queue_len": 0, 2024-09-13T12:11:07.686 INFO:tasks.workunit.client.0.smithi032.stdout: "num_snap_trimming": 0, 2024-09-13T12:11:07.686 INFO:tasks.workunit.client.0.smithi032.stdout: "num_shards_repaired": 0, 2024-09-13T12:11:07.686 INFO:tasks.workunit.client.0.smithi032.stdout: "op_queue_age_hist": { 2024-09-13T12:11:07.686 INFO:tasks.workunit.client.0.smithi032.stdout: "histogram": [], 2024-09-13T12:11:07.686 INFO:tasks.workunit.client.0.smithi032.stdout: "upper_bound": 1 2024-09-13T12:11:07.686 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.686 INFO:tasks.workunit.client.0.smithi032.stdout: "perf_stat": { 2024-09-13T12:11:07.686 INFO:tasks.workunit.client.0.smithi032.stdout: "commit_latency_ms": 85, 2024-09-13T12:11:07.686 INFO:tasks.workunit.client.0.smithi032.stdout: "apply_latency_ms": 85, 2024-09-13T12:11:07.686 INFO:tasks.workunit.client.0.smithi032.stdout: "commit_latency_ns": 85000000, 2024-09-13T12:11:07.686 INFO:tasks.workunit.client.0.smithi032.stdout: "apply_latency_ns": 85000000 2024-09-13T12:11:07.687 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.687 INFO:tasks.workunit.client.0.smithi032.stdout: "alerts": [], 2024-09-13T12:11:07.687 INFO:tasks.workunit.client.0.smithi032.stdout: "network_ping_times": [] 2024-09-13T12:11:07.687 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.687 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.687 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_stats": [ 2024-09-13T12:11:07.687 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.687 INFO:tasks.workunit.client.0.smithi032.stdout: "poolid": 1, 2024-09-13T12:11:07.687 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg": 32, 2024-09-13T12:11:07.687 INFO:tasks.workunit.client.0.smithi032.stdout: "stat_sum": { 2024-09-13T12:11:07.687 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes": 48843, 2024-09-13T12:11:07.687 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects": 103, 2024-09-13T12:11:07.688 INFO:tasks.workunit.client.0.smithi032.stdout: "num_object_clones": 0, 2024-09-13T12:11:07.688 INFO:tasks.workunit.client.0.smithi032.stdout: "num_object_copies": 309, 2024-09-13T12:11:07.688 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_missing_on_primary": 0, 2024-09-13T12:11:07.688 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_missing": 0, 2024-09-13T12:11:07.688 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_degraded": 103, 2024-09-13T12:11:07.688 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_misplaced": 0, 2024-09-13T12:11:07.688 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_unfound": 0, 2024-09-13T12:11:07.688 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_dirty": 103, 2024-09-13T12:11:07.688 INFO:tasks.workunit.client.0.smithi032.stdout: "num_whiteouts": 0, 2024-09-13T12:11:07.688 INFO:tasks.workunit.client.0.smithi032.stdout: "num_read": 0, 2024-09-13T12:11:07.688 INFO:tasks.workunit.client.0.smithi032.stdout: "num_read_kb": 0, 2024-09-13T12:11:07.688 INFO:tasks.workunit.client.0.smithi032.stdout: "num_write": 57, 2024-09-13T12:11:07.688 INFO:tasks.workunit.client.0.smithi032.stdout: "num_write_kb": 57, 2024-09-13T12:11:07.689 INFO:tasks.workunit.client.0.smithi032.stdout: "num_scrub_errors": 0, 2024-09-13T12:11:07.689 INFO:tasks.workunit.client.0.smithi032.stdout: "num_shallow_scrub_errors": 0, 2024-09-13T12:11:07.689 INFO:tasks.workunit.client.0.smithi032.stdout: "num_deep_scrub_errors": 0, 2024-09-13T12:11:07.689 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_recovered": 0, 2024-09-13T12:11:07.689 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes_recovered": 0, 2024-09-13T12:11:07.689 INFO:tasks.workunit.client.0.smithi032.stdout: "num_keys_recovered": 0, 2024-09-13T12:11:07.689 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_omap": 0, 2024-09-13T12:11:07.689 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_hit_set_archive": 0, 2024-09-13T12:11:07.689 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes_hit_set_archive": 0, 2024-09-13T12:11:07.689 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush": 0, 2024-09-13T12:11:07.689 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_kb": 0, 2024-09-13T12:11:07.689 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict": 0, 2024-09-13T12:11:07.689 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_kb": 0, 2024-09-13T12:11:07.690 INFO:tasks.workunit.client.0.smithi032.stdout: "num_promote": 0, 2024-09-13T12:11:07.690 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_mode_high": 0, 2024-09-13T12:11:07.690 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_mode_low": 0, 2024-09-13T12:11:07.690 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_mode_some": 0, 2024-09-13T12:11:07.690 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_mode_full": 0, 2024-09-13T12:11:07.690 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_pinned": 0, 2024-09-13T12:11:07.690 INFO:tasks.workunit.client.0.smithi032.stdout: "num_legacy_snapsets": 0, 2024-09-13T12:11:07.690 INFO:tasks.workunit.client.0.smithi032.stdout: "num_large_omap_objects": 0, 2024-09-13T12:11:07.690 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_manifest": 0, 2024-09-13T12:11:07.690 INFO:tasks.workunit.client.0.smithi032.stdout: "num_omap_bytes": 0, 2024-09-13T12:11:07.690 INFO:tasks.workunit.client.0.smithi032.stdout: "num_omap_keys": 0, 2024-09-13T12:11:07.690 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_repaired": 0 2024-09-13T12:11:07.690 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.690 INFO:tasks.workunit.client.0.smithi032.stdout: "store_stats": { 2024-09-13T12:11:07.691 INFO:tasks.workunit.client.0.smithi032.stdout: "total": 0, 2024-09-13T12:11:07.691 INFO:tasks.workunit.client.0.smithi032.stdout: "available": 0, 2024-09-13T12:11:07.691 INFO:tasks.workunit.client.0.smithi032.stdout: "internally_reserved": 0, 2024-09-13T12:11:07.691 INFO:tasks.workunit.client.0.smithi032.stdout: "allocated": 1220608, 2024-09-13T12:11:07.691 INFO:tasks.workunit.client.0.smithi032.stdout: "data_stored": 143276, 2024-09-13T12:11:07.691 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed": 0, 2024-09-13T12:11:07.691 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_allocated": 0, 2024-09-13T12:11:07.691 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_original": 0, 2024-09-13T12:11:07.691 INFO:tasks.workunit.client.0.smithi032.stdout: "omap_allocated": 0, 2024-09-13T12:11:07.691 INFO:tasks.workunit.client.0.smithi032.stdout: "internal_metadata": 0 2024-09-13T12:11:07.691 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.691 INFO:tasks.workunit.client.0.smithi032.stdout: "log_size": 473, 2024-09-13T12:11:07.691 INFO:tasks.workunit.client.0.smithi032.stdout: "ondisk_log_size": 473, 2024-09-13T12:11:07.692 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 64, 2024-09-13T12:11:07.692 INFO:tasks.workunit.client.0.smithi032.stdout: "acting": 64, 2024-09-13T12:11:07.692 INFO:tasks.workunit.client.0.smithi032.stdout: "num_store_stats": 3 2024-09-13T12:11:07.692 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.692 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:07.692 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_stats": [ 2024-09-13T12:11:07.692 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.692 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 0, 2024-09-13T12:11:07.692 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 270582939659 2024-09-13T12:11:07.692 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.692 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.692 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 1, 2024-09-13T12:11:07.693 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 42949672988 2024-09-13T12:11:07.693 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.693 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.693 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2, 2024-09-13T12:11:07.693 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 279172874250 2024-09-13T12:11:07.693 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.693 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:07.693 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg_by_state": [ 2024-09-13T12:11:07.693 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.693 INFO:tasks.workunit.client.0.smithi032.stdout: "state": "active+undersized+degraded", 2024-09-13T12:11:07.693 INFO:tasks.workunit.client.0.smithi032.stdout: "num": 32 2024-09-13T12:11:07.693 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.693 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:07.694 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg_by_osd": [ 2024-09-13T12:11:07.694 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.694 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2, 2024-09-13T12:11:07.694 INFO:tasks.workunit.client.0.smithi032.stdout: "num_primary_pg": 18, 2024-09-13T12:11:07.694 INFO:tasks.workunit.client.0.smithi032.stdout: "num_acting_pg": 32, 2024-09-13T12:11:07.694 INFO:tasks.workunit.client.0.smithi032.stdout: "num_up_not_acting_pg": 0 2024-09-13T12:11:07.694 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.694 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.694 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 0, 2024-09-13T12:11:07.694 INFO:tasks.workunit.client.0.smithi032.stdout: "num_primary_pg": 14, 2024-09-13T12:11:07.695 INFO:tasks.workunit.client.0.smithi032.stdout: "num_acting_pg": 32, 2024-09-13T12:11:07.695 INFO:tasks.workunit.client.0.smithi032.stdout: "num_up_not_acting_pg": 0 2024-09-13T12:11:07.695 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.695 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.695 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 1, 2024-09-13T12:11:07.695 INFO:tasks.workunit.client.0.smithi032.stdout: "num_primary_pg": 0, 2024-09-13T12:11:07.695 INFO:tasks.workunit.client.0.smithi032.stdout: "num_acting_pg": 0, 2024-09-13T12:11:07.695 INFO:tasks.workunit.client.0.smithi032.stdout: "num_up_not_acting_pg": 0 2024-09-13T12:11:07.695 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.695 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:07.695 INFO:tasks.workunit.client.0.smithi032.stdout: "purged_snaps": [ 2024-09-13T12:11:07.696 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:07.696 INFO:tasks.workunit.client.0.smithi032.stdout: "pool": 1, 2024-09-13T12:11:07.696 INFO:tasks.workunit.client.0.smithi032.stdout: "purged_snaps": {} 2024-09-13T12:11:07.696 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.696 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:07.696 INFO:tasks.workunit.client.0.smithi032.stdout: "servicemap": { 2024-09-13T12:11:07.696 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 5, 2024-09-13T12:11:07.696 INFO:tasks.workunit.client.0.smithi032.stdout: "modified": "2024-09-13T12:10:22.374555+0000", 2024-09-13T12:11:07.696 INFO:tasks.workunit.client.0.smithi032.stdout: "services": {} 2024-09-13T12:11:07.696 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:07.696 INFO:tasks.workunit.client.0.smithi032.stdout: "mgrstat_first_committed": 1, 2024-09-13T12:11:07.696 INFO:tasks.workunit.client.0.smithi032.stdout: "mgrstat_last_committed": 95, 2024-09-13T12:11:07.696 INFO:tasks.workunit.client.0.smithi032.stdout: "logm_first_committed": 1, 2024-09-13T12:11:07.696 INFO:tasks.workunit.client.0.smithi032.stdout: "logm_last_committed": 431, 2024-09-13T12:11:07.697 INFO:tasks.workunit.client.0.smithi032.stdout: "paxos": { 2024-09-13T12:11:07.697 INFO:tasks.workunit.client.0.smithi032.stdout: "first_committed": 1, 2024-09-13T12:11:07.697 INFO:tasks.workunit.client.0.smithi032.stdout: "last_committed": 591, 2024-09-13T12:11:07.697 INFO:tasks.workunit.client.0.smithi032.stdout: "last_pn": 0, 2024-09-13T12:11:07.697 INFO:tasks.workunit.client.0.smithi032.stdout: "accepted_pn": 0 2024-09-13T12:11:07.697 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:07.697 INFO:tasks.workunit.client.0.smithi032.stdout:} 2024-09-13T12:11:07.698 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-13T12:11:07.698 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: expr 2 + 2 2024-09-13T12:11:07.698 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: objname=existing_4 2024-09-13T12:11:07.698 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2024-09-13T12:11:07.699 INFO:tasks.workunit.client.0.smithi032.stdout:writing non-divergent object existing_4 2024-09-13T12:11:07.699 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2024-09-13T12:11:07.969 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:11:07.970 INFO:tasks.workunit.client.0.smithi032.stdout:1.1f 3 0 3 0 0 1615 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422575+0000 38'45 73:109 [2,0] 2 [2,0] 2 38'45 2024-09-13T12:09:53.714380+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:07.970 INFO:tasks.workunit.client.0.smithi032.stdout:1.1e 3 0 3 0 0 1615 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.291074+0000 44'45 73:117 [0,2] 0 [0,2] 0 44'45 2024-09-13T12:10:01.695428+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:07.970 INFO:tasks.workunit.client.0.smithi032.stdout:1.1d 4 0 4 0 0 1932 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:32.422167+0000 39'46 73:121 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:49.397412+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T15:29:37.811752+0000 2 0 2024-09-13T12:11:07.970 INFO:tasks.workunit.client.0.smithi032.stdout:1.b 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.290838+0000 42'44 73:117 [0,2] 0 [0,2] 0 42'44 2024-09-13T12:09:57.774856+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:10:36.425691+0000 0 0 2024-09-13T12:11:07.970 INFO:tasks.workunit.client.0.smithi032.stdout:1.a 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422529+0000 34'44 73:92 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:50.853058+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:07.970 INFO:tasks.workunit.client.0.smithi032.stdout:1.9 5 0 5 0 0 2415 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.036386+0000 42'45 73:132 [2,0] 2 [2,0] 2 42'45 2024-09-13T12:10:03.691408+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T17:45:19.630728+0000 4 0 2024-09-13T12:11:07.970 INFO:tasks.workunit.client.0.smithi032.stdout:1.8 5 0 5 0 0 2098 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:32.422809+0000 42'46 73:115 [2,0] 2 [2,0] 2 42'46 2024-09-13T12:09:58.652148+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:07.970 INFO:tasks.workunit.client.0.smithi032.stdout:1.7 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422184+0000 34'44 73:109 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:55.719570+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:07.970 INFO:tasks.workunit.client.0.smithi032.stdout:1.6 5 0 5 0 0 2098 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291701+0000 41'46 73:130 [0,2] 0 [0,2] 0 41'46 2024-09-13T12:10:02.648875+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:07.970 INFO:tasks.workunit.client.0.smithi032.stdout:1.5 6 0 6 0 0 2898 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:32.422143+0000 42'46 73:118 [2,0] 2 [2,0] 2 38'45 2024-09-13T12:09:52.375427+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T21:28:19.785529+0000 5 0 2024-09-13T12:11:07.970 INFO:tasks.workunit.client.0.smithi032.stdout:1.4 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.290860+0000 34'44 73:100 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:46.701189+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:07.971 INFO:tasks.workunit.client.0.smithi032.stdout:1.3 5 0 5 0 0 2415 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.363328+0000 44'45 73:123 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:44.638103+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T20:31:20.891228+0000 3 0 2024-09-13T12:11:07.971 INFO:tasks.workunit.client.0.smithi032.stdout:1.2 1 0 1 0 0 483 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.291679+0000 39'44 73:118 [0,2] 0 [0,2] 0 39'44 2024-09-13T12:09:55.946588+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T18:12:36.231144+0000 1 0 2024-09-13T12:11:07.971 INFO:tasks.workunit.client.0.smithi032.stdout:1.c 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.290976+0000 41'45 73:127 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:51.776896+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:19:05.496290+0000 2 0 2024-09-13T12:11:07.971 INFO:tasks.workunit.client.0.smithi032.stdout:1.1 2 0 2 0 0 966 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422961+0000 41'45 73:114 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:48.455862+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:18:33.965292+0000 0 0 2024-09-13T12:11:07.971 INFO:tasks.workunit.client.0.smithi032.stdout:1.1c 2 0 2 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.291485+0000 34'44 73:99 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:49.691404+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:07.971 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 3 0 3 0 0 1449 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.291469+0000 42'44 73:129 [0,2] 0 [0,2] 0 32'43 2024-09-13T12:09:43.710370+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:01:49.441566+0000 2 0 2024-09-13T12:11:07.971 INFO:tasks.workunit.client.0.smithi032.stdout:1.d 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422420+0000 42'45 73:117 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:50.728150+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:24:12.357955+0000 1 0 2024-09-13T12:11:07.971 INFO:tasks.workunit.client.0.smithi032.stdout:1.e 4 0 4 0 0 1932 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.291323+0000 42'45 73:120 [0,2] 0 [0,2] 0 32'43 2024-09-13T12:09:49.907160+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:38:43.613716+0000 2 0 2024-09-13T12:11:07.971 INFO:tasks.workunit.client.0.smithi032.stdout:1.f 2 0 2 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.363578+0000 38'44 73:97 [2,0] 2 [2,0] 2 38'44 2024-09-13T12:09:59.616191+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:07.971 INFO:tasks.workunit.client.0.smithi032.stdout:1.10 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.291344+0000 43'45 73:133 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:52.759350+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T12:46:25.509739+0000 2 0 2024-09-13T12:11:07.972 INFO:tasks.workunit.client.0.smithi032.stdout:1.11 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.363401+0000 34'45 73:124 [2,0] 2 [2,0] 2 34'45 2024-09-13T12:09:53.421579+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T15:04:47.312019+0000 3 0 2024-09-13T12:11:07.972 INFO:tasks.workunit.client.0.smithi032.stdout:1.12 1 0 1 0 0 483 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.290912+0000 34'44 73:123 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:52.858658+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T23:50:26.431309+0000 1 0 2024-09-13T12:11:07.972 INFO:tasks.workunit.client.0.smithi032.stdout:1.13 3 0 3 0 0 1615 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.939197+0000 44'45 73:120 [2,0] 2 [2,0] 2 44'45 2024-09-13T12:10:04.717933+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:10:37.072583+0000 0 0 2024-09-13T12:11:07.972 INFO:tasks.workunit.client.0.smithi032.stdout:1.14 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.291845+0000 44'45 73:135 [0,2] 0 [0,2] 0 32'43 2024-09-13T12:09:50.727967+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T18:54:37.906181+0000 1 0 2024-09-13T12:11:07.972 INFO:tasks.workunit.client.0.smithi032.stdout:1.15 3 0 3 0 0 1449 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422194+0000 38'44 73:112 [2,0] 2 [2,0] 2 38'44 2024-09-13T12:09:54.402988+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:26:56.754038+0000 3 0 2024-09-13T12:11:07.972 INFO:tasks.workunit.client.0.smithi032.stdout:1.16 4 0 4 0 0 1932 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291960+0000 38'46 73:138 [0,2] 0 [0,2] 0 38'46 2024-09-13T12:10:00.648885+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:11:39.586367+0000 4 0 2024-09-13T12:11:07.972 INFO:tasks.workunit.client.0.smithi032.stdout:1.17 1 0 1 0 0 649 0 0 13 30 13 active+undersized+degraded 2024-09-13T12:10:32.423246+0000 32'43 73:96 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:56.659002+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:07.972 INFO:tasks.workunit.client.0.smithi032.stdout:1.18 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422567+0000 44'45 73:127 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:47.765518+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T12:54:37.159964+0000 1 0 2024-09-13T12:11:07.972 INFO:tasks.workunit.client.0.smithi032.stdout:1.19 2 0 2 0 0 966 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422893+0000 34'44 73:139 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:51.375338+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:16:57.418519+0000 2 0 2024-09-13T12:11:07.972 INFO:tasks.workunit.client.0.smithi032.stdout:1.1a 4 0 4 0 0 1932 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.423351+0000 41'45 73:119 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:51.953415+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T21:57:52.925654+0000 3 0 2024-09-13T12:11:07.972 INFO:tasks.workunit.client.0.smithi032.stdout:1.1b 5 0 5 0 0 2415 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291329+0000 44'46 73:140 [0,2] 0 [0,2] 0 36'45 2024-09-13T12:09:54.677751+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T15:59:12.687663+0000 4 0 2024-09-13T12:11:07.972 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:11:07.973 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:11:07.973 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:11:07.981 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2024-09-13T12:11:08.070 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-13T12:11:08.070 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: wait_for_clean 2024-09-13T12:11:08.070 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T12:11:08.070 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T12:11:08.071 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T12:11:08.071 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-13T12:11:08.071 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T12:11:08.071 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T12:11:08.071 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T12:11:08.072 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T12:11:08.072 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T12:11:08.127 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-13T12:11:08.127 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T12:11:08.127 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T12:11:08.127 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T12:11:08.128 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T12:11:08.128 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T12:11:08.589 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T12:11:08.589 INFO:tasks.workunit.client.0.smithi032.stderr:1 2024-09-13T12:11:08.589 INFO:tasks.workunit.client.0.smithi032.stderr:2' 2024-09-13T12:11:08.589 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T12:11:08.589 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:11:08.589 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T12:11:08.741 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=270582939661 2024-09-13T12:11:08.741 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 270582939661 2024-09-13T12:11:08.741 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-270582939661' 2024-09-13T12:11:08.741 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:11:08.741 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T12:11:08.832 INFO:tasks.workunit.client.0.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-13T12:11:08.834 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-13T12:11:08.834 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-13T12:11:08.834 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-13T12:11:08.834 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:11:08.834 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T12:11:08.959 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=279172874253 2024-09-13T12:11:08.959 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 279172874253 2024-09-13T12:11:08.959 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-270582939661 2-279172874253' 2024-09-13T12:11:08.959 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:11:08.959 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-270582939661 2024-09-13T12:11:08.959 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:11:08.960 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T12:11:08.960 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-270582939661 2024-09-13T12:11:08.961 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:11:08.961 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=270582939661 2024-09-13T12:11:08.962 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 270582939661' 2024-09-13T12:11:08.962 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.0 seq 270582939661 2024-09-13T12:11:08.962 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:11:09.408 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 270582939659 -lt 270582939661 2024-09-13T12:11:09.409 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T12:11:10.409 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T12:11:10.410 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:11:10.859 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 270582939662 -lt 270582939661 2024-09-13T12:11:10.859 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:11:10.859 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-279172874253 2024-09-13T12:11:10.860 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:11:10.860 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T12:11:10.861 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-279172874253 2024-09-13T12:11:10.861 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:11:10.862 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=279172874253 2024-09-13T12:11:10.862 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.2 seq 279172874253 2024-09-13T12:11:10.862 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 279172874253' 2024-09-13T12:11:10.862 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T12:11:11.302 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 279172874253 -lt 279172874253 2024-09-13T12:11:11.303 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T12:11:11.303 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:11:11.303 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:11:11.857 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 32 == 0 2024-09-13T12:11:11.857 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:11:11.858 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:11:11.858 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:11:11.858 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:11:11.859 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:11:11.859 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:11:11.859 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:11:12.271 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:11:12.272 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:11:12.272 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:11:12.272 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:11:12.834 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-13T12:11:12.835 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-13T12:11:12.835 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T12:11:12.835 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-13T12:11:12.835 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:11:12.835 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T12:11:12.936 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:11:12.936 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:11:12.936 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:11:12.936 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:11:12.936 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:11:12.936 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:11:12.936 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:11:12.937 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:11:13.339 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:11:13.339 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:11:13.339 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:11:13.339 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:11:13.900 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-13T12:11:13.900 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:11:13.900 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:11:13.900 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:11:13.900 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:11:13.900 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:11:13.900 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:11:13.900 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:11:13.900 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:11:14.464 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:11:14.464 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:11:14.464 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-13T12:11:14.464 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:11:14.464 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T12:11:14.665 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:11:14.666 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:11:14.666 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:11:14.666 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:11:14.666 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:11:14.666 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:11:14.666 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:11:14.666 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:11:15.066 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:11:15.066 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:11:15.066 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:11:15.066 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:11:15.616 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-13T12:11:15.617 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:11:15.617 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:11:15.617 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:11:15.617 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:11:15.617 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:11:15.617 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:11:15.617 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:11:15.617 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:11:16.199 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:11:16.199 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:11:16.199 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-13T12:11:16.199 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:11:16.199 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-13T12:11:16.601 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:11:16.601 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:11:16.601 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:11:16.601 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:11:16.601 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:11:16.601 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:11:16.601 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:11:16.602 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:11:17.004 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:11:17.004 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:11:17.004 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:11:17.004 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:11:17.555 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-13T12:11:17.555 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:11:17.556 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:11:17.556 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:11:17.556 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:11:17.556 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:11:17.556 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:11:17.556 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:11:17.556 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:11:18.121 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:11:18.121 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:11:18.121 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-13T12:11:18.121 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:11:18.121 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-13T12:11:18.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:11:18.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:11:18.923 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:11:18.923 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:11:18.923 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:11:18.923 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:11:18.923 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:11:18.923 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:11:19.336 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:11:19.336 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:11:19.337 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:11:19.337 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:11:19.895 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-13T12:11:19.895 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:11:19.895 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:11:19.895 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:11:19.895 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:11:19.895 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:11:19.896 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:11:19.896 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:11:19.896 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:11:20.457 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:11:20.457 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:11:20.457 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-13T12:11:20.458 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:11:20.458 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-13T12:11:22.058 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:11:22.058 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:11:22.059 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:11:22.059 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:11:22.059 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:11:22.059 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:11:22.059 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:11:22.059 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:11:22.476 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:11:22.476 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:11:22.476 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:11:22.477 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:11:23.032 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-13T12:11:23.032 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:11:23.032 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:11:23.032 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:11:23.033 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:11:23.033 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:11:23.033 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:11:23.033 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:11:23.033 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:11:23.616 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:11:23.616 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:11:23.616 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-13T12:11:23.616 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:11:23.616 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-13T12:11:26.817 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:11:26.817 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:11:26.817 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:11:26.818 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:11:26.818 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:11:26.818 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:11:26.818 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:11:26.818 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:11:27.233 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:11:27.233 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:11:27.233 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:11:27.234 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:11:27.806 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-13T12:11:27.806 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:11:27.806 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:11:27.806 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:11:27.806 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:11:27.806 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:11:27.806 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:11:27.806 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:11:27.807 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:11:28.377 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:11:28.377 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:11:28.378 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-13T12:11:28.378 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:11:28.378 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-13T12:11:34.779 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:11:34.779 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:11:34.779 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:11:34.780 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:11:34.780 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:11:34.780 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:11:34.780 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:11:34.780 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:11:35.194 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:11:35.194 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:11:35.194 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:11:35.195 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:11:35.748 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-13T12:11:35.749 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:11:35.749 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:11:35.749 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:11:35.749 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:11:35.749 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:11:35.749 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:11:35.749 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:11:35.749 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:11:36.341 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:11:36.341 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:11:36.341 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-13T12:11:36.341 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:11:36.341 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-13T12:11:43.642 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:11:43.643 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:11:43.643 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:11:43.643 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:11:43.643 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:11:43.643 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:11:43.643 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:11:43.643 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:11:44.057 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:11:44.058 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:11:44.058 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:11:44.058 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:11:44.617 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-13T12:11:44.617 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:11:44.618 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:11:44.618 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:11:44.618 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:11:44.618 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:11:44.618 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:11:44.618 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:11:44.618 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:11:45.198 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:11:45.198 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:11:45.198 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-13T12:11:45.198 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-13T12:11:45.755 INFO:tasks.workunit.client.0.smithi032.stdout:{ 2024-09-13T12:11:45.755 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_fingerprint": "30b75177-5678-43f5-989c-4de966e73592", 2024-09-13T12:11:45.755 INFO:tasks.workunit.client.0.smithi032.stdout: "version": "19.3.0-4697-gd58f7306", 2024-09-13T12:11:45.755 INFO:tasks.workunit.client.0.smithi032.stdout: "commit": "d58f73069a8157a9ace12a0247ee3e8497707ea9", 2024-09-13T12:11:45.756 INFO:tasks.workunit.client.0.smithi032.stdout: "timestamp": "2024-09-13T12:11:45.754149+0000", 2024-09-13T12:11:45.756 INFO:tasks.workunit.client.0.smithi032.stdout: "tag": "", 2024-09-13T12:11:45.756 INFO:tasks.workunit.client.0.smithi032.stdout: "health": { 2024-09-13T12:11:45.756 INFO:tasks.workunit.client.0.smithi032.stdout: "status": "HEALTH_WARN", 2024-09-13T12:11:45.756 INFO:tasks.workunit.client.0.smithi032.stdout: "checks": { 2024-09-13T12:11:45.756 INFO:tasks.workunit.client.0.smithi032.stdout: "OSDMAP_FLAGS": { 2024-09-13T12:11:45.756 INFO:tasks.workunit.client.0.smithi032.stdout: "severity": "HEALTH_WARN", 2024-09-13T12:11:45.756 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": { 2024-09-13T12:11:45.756 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-13T12:11:45.756 INFO:tasks.workunit.client.0.smithi032.stdout: "count": 17 2024-09-13T12:11:45.756 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.757 INFO:tasks.workunit.client.0.smithi032.stdout: "detail": [], 2024-09-13T12:11:45.757 INFO:tasks.workunit.client.0.smithi032.stdout: "muted": false 2024-09-13T12:11:45.757 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.757 INFO:tasks.workunit.client.0.smithi032.stdout: "OSD_DOWN": { 2024-09-13T12:11:45.757 INFO:tasks.workunit.client.0.smithi032.stdout: "severity": "HEALTH_WARN", 2024-09-13T12:11:45.757 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": { 2024-09-13T12:11:45.757 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "1 osds down", 2024-09-13T12:11:45.757 INFO:tasks.workunit.client.0.smithi032.stdout: "count": 1 2024-09-13T12:11:45.757 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.757 INFO:tasks.workunit.client.0.smithi032.stdout: "detail": [ 2024-09-13T12:11:45.758 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.758 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "osd.1 (root=default,host=smithi032) is down" 2024-09-13T12:11:45.758 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.758 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:45.758 INFO:tasks.workunit.client.0.smithi032.stdout: "muted": false 2024-09-13T12:11:45.758 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.758 INFO:tasks.workunit.client.0.smithi032.stdout: "PG_DEGRADED": { 2024-09-13T12:11:45.758 INFO:tasks.workunit.client.0.smithi032.stdout: "severity": "HEALTH_WARN", 2024-09-13T12:11:45.758 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": { 2024-09-13T12:11:45.758 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "Degraded data redundancy: 103/309 objects degraded (33.333%), 32 pgs degraded, 32 pgs undersized", 2024-09-13T12:11:45.758 INFO:tasks.workunit.client.0.smithi032.stdout: "count": 64 2024-09-13T12:11:45.759 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.759 INFO:tasks.workunit.client.0.smithi032.stdout: "detail": [ 2024-09-13T12:11:45.759 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.759 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.0 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-13T12:11:45.759 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.759 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.759 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.1 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-13T12:11:45.759 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.759 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.759 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.2 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-13T12:11:45.759 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.760 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.760 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.3 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-13T12:11:45.760 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.760 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.760 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.4 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-13T12:11:45.760 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.760 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.760 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.5 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-13T12:11:45.760 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.760 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.760 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.6 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-13T12:11:45.760 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.761 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.761 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.7 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-13T12:11:45.761 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.761 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.761 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.8 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-13T12:11:45.761 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.761 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.761 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.9 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-13T12:11:45.761 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.761 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.762 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.a is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-13T12:11:45.762 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.762 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.762 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.b is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-13T12:11:45.762 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.762 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.762 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.c is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-13T12:11:45.762 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.762 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.762 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.d is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-13T12:11:45.763 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.763 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.763 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.e is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-13T12:11:45.763 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.763 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.763 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.f is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-13T12:11:45.763 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.763 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.763 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.10 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-13T12:11:45.763 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.763 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.764 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.11 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-13T12:11:45.764 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.764 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.764 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.12 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-13T12:11:45.764 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.764 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.764 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.13 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-13T12:11:45.764 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.764 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.765 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.14 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-13T12:11:45.765 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.765 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.765 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.15 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-13T12:11:45.765 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.765 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.765 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.16 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-13T12:11:45.765 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.765 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.765 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.17 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-13T12:11:45.765 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.765 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.766 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.18 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-13T12:11:45.766 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.766 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.766 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.19 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-13T12:11:45.766 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.766 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.766 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.1a is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-13T12:11:45.766 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.767 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.767 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.1b is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-13T12:11:45.767 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.767 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.767 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.1c is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-13T12:11:45.767 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.767 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.767 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.1d is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-13T12:11:45.767 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.767 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.767 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.1e is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-13T12:11:45.768 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.768 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.768 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.1f is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-13T12:11:45.768 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.768 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:45.768 INFO:tasks.workunit.client.0.smithi032.stdout: "muted": false 2024-09-13T12:11:45.768 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.768 INFO:tasks.workunit.client.0.smithi032.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-13T12:11:45.768 INFO:tasks.workunit.client.0.smithi032.stdout: "severity": "HEALTH_WARN", 2024-09-13T12:11:45.768 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": { 2024-09-13T12:11:45.769 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-13T12:11:45.769 INFO:tasks.workunit.client.0.smithi032.stdout: "count": 1 2024-09-13T12:11:45.769 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.769 INFO:tasks.workunit.client.0.smithi032.stdout: "detail": [ 2024-09-13T12:11:45.769 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.769 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "application not enabled on pool 'test'" 2024-09-13T12:11:45.769 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.769 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.769 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-13T12:11:45.769 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.769 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:45.769 INFO:tasks.workunit.client.0.smithi032.stdout: "muted": false 2024-09-13T12:11:45.770 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.770 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.770 INFO:tasks.workunit.client.0.smithi032.stdout: "mutes": [] 2024-09-13T12:11:45.770 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.770 INFO:tasks.workunit.client.0.smithi032.stdout: "monmap_first_committed": 1, 2024-09-13T12:11:45.770 INFO:tasks.workunit.client.0.smithi032.stdout: "monmap_last_committed": 1, 2024-09-13T12:11:45.770 INFO:tasks.workunit.client.0.smithi032.stdout: "monmap": { 2024-09-13T12:11:45.770 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 1, 2024-09-13T12:11:45.770 INFO:tasks.workunit.client.0.smithi032.stdout: "fsid": "e12dfbc3-9bf5-45db-a6f0-f25db93ee351", 2024-09-13T12:11:45.770 INFO:tasks.workunit.client.0.smithi032.stdout: "modified": "2024-09-13T12:07:34.499882Z", 2024-09-13T12:11:45.770 INFO:tasks.workunit.client.0.smithi032.stdout: "created": "2024-09-13T12:07:34.499882Z", 2024-09-13T12:11:45.771 INFO:tasks.workunit.client.0.smithi032.stdout: "min_mon_release": 19, 2024-09-13T12:11:45.771 INFO:tasks.workunit.client.0.smithi032.stdout: "min_mon_release_name": "squid", 2024-09-13T12:11:45.771 INFO:tasks.workunit.client.0.smithi032.stdout: "election_strategy": 1, 2024-09-13T12:11:45.771 INFO:tasks.workunit.client.0.smithi032.stdout: "disallowed_leaders: ": "", 2024-09-13T12:11:45.771 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_mode": false, 2024-09-13T12:11:45.771 INFO:tasks.workunit.client.0.smithi032.stdout: "tiebreaker_mon": "", 2024-09-13T12:11:45.771 INFO:tasks.workunit.client.0.smithi032.stdout: "removed_ranks: ": "", 2024-09-13T12:11:45.771 INFO:tasks.workunit.client.0.smithi032.stdout: "features": { 2024-09-13T12:11:45.771 INFO:tasks.workunit.client.0.smithi032.stdout: "persistent": [ 2024-09-13T12:11:45.771 INFO:tasks.workunit.client.0.smithi032.stdout: "kraken", 2024-09-13T12:11:45.771 INFO:tasks.workunit.client.0.smithi032.stdout: "luminous", 2024-09-13T12:11:45.771 INFO:tasks.workunit.client.0.smithi032.stdout: "mimic", 2024-09-13T12:11:45.771 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap-prune", 2024-09-13T12:11:45.772 INFO:tasks.workunit.client.0.smithi032.stdout: "nautilus", 2024-09-13T12:11:45.772 INFO:tasks.workunit.client.0.smithi032.stdout: "octopus", 2024-09-13T12:11:45.772 INFO:tasks.workunit.client.0.smithi032.stdout: "pacific", 2024-09-13T12:11:45.772 INFO:tasks.workunit.client.0.smithi032.stdout: "elector-pinging", 2024-09-13T12:11:45.772 INFO:tasks.workunit.client.0.smithi032.stdout: "quincy", 2024-09-13T12:11:45.772 INFO:tasks.workunit.client.0.smithi032.stdout: "reef", 2024-09-13T12:11:45.772 INFO:tasks.workunit.client.0.smithi032.stdout: "squid" 2024-09-13T12:11:45.772 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:45.772 INFO:tasks.workunit.client.0.smithi032.stdout: "optional": [] 2024-09-13T12:11:45.772 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.772 INFO:tasks.workunit.client.0.smithi032.stdout: "mons": [ 2024-09-13T12:11:45.773 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.773 INFO:tasks.workunit.client.0.smithi032.stdout: "rank": 0, 2024-09-13T12:11:45.773 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "a", 2024-09-13T12:11:45.773 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addrs": { 2024-09-13T12:11:45.773 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:11:45.773 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.773 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:11:45.773 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:7115", 2024-09-13T12:11:45.773 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 0 2024-09-13T12:11:45.773 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.773 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:45.773 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.774 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:7115/0", 2024-09-13T12:11:45.774 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-13T12:11:45.774 INFO:tasks.workunit.client.0.smithi032.stdout: "priority": 0, 2024-09-13T12:11:45.774 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 0, 2024-09-13T12:11:45.774 INFO:tasks.workunit.client.0.smithi032.stdout: "crush_location": "{}" 2024-09-13T12:11:45.774 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.774 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:45.774 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.774 INFO:tasks.workunit.client.0.smithi032.stdout: "quorum": [ 2024-09-13T12:11:45.774 INFO:tasks.workunit.client.0.smithi032.stdout: 0 2024-09-13T12:11:45.774 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:45.775 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap": { 2024-09-13T12:11:45.775 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 73, 2024-09-13T12:11:45.775 INFO:tasks.workunit.client.0.smithi032.stdout: "fsid": "e12dfbc3-9bf5-45db-a6f0-f25db93ee351", 2024-09-13T12:11:45.775 INFO:tasks.workunit.client.0.smithi032.stdout: "created": "2024-09-13T12:07:35.402712+0000", 2024-09-13T12:11:45.775 INFO:tasks.workunit.client.0.smithi032.stdout: "modified": "2024-09-13T12:10:31.792953+0000", 2024-09-13T12:11:45.775 INFO:tasks.workunit.client.0.smithi032.stdout: "last_up_change": "2024-09-13T12:10:28.520213+0000", 2024-09-13T12:11:45.775 INFO:tasks.workunit.client.0.smithi032.stdout: "last_in_change": "2024-09-13T12:10:30.500123+0000", 2024-09-13T12:11:45.775 INFO:tasks.workunit.client.0.smithi032.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T12:11:45.775 INFO:tasks.workunit.client.0.smithi032.stdout: "flags_num": 5800384, 2024-09-13T12:11:45.775 INFO:tasks.workunit.client.0.smithi032.stdout: "flags_set": [ 2024-09-13T12:11:45.775 INFO:tasks.workunit.client.0.smithi032.stdout: "nodown", 2024-09-13T12:11:45.775 INFO:tasks.workunit.client.0.smithi032.stdout: "noin", 2024-09-13T12:11:45.775 INFO:tasks.workunit.client.0.smithi032.stdout: "noout", 2024-09-13T12:11:45.776 INFO:tasks.workunit.client.0.smithi032.stdout: "pglog_hardlimit", 2024-09-13T12:11:45.776 INFO:tasks.workunit.client.0.smithi032.stdout: "purged_snapdirs", 2024-09-13T12:11:45.776 INFO:tasks.workunit.client.0.smithi032.stdout: "recovery_deletes", 2024-09-13T12:11:45.776 INFO:tasks.workunit.client.0.smithi032.stdout: "sortbitwise" 2024-09-13T12:11:45.776 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:45.776 INFO:tasks.workunit.client.0.smithi032.stdout: "crush_version": 9, 2024-09-13T12:11:45.776 INFO:tasks.workunit.client.0.smithi032.stdout: "full_ratio": 0.99000000953674316, 2024-09-13T12:11:45.776 INFO:tasks.workunit.client.0.smithi032.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-13T12:11:45.776 INFO:tasks.workunit.client.0.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T12:11:45.776 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_snapshot": "", 2024-09-13T12:11:45.776 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_max": 1, 2024-09-13T12:11:45.776 INFO:tasks.workunit.client.0.smithi032.stdout: "max_osd": 3, 2024-09-13T12:11:45.777 INFO:tasks.workunit.client.0.smithi032.stdout: "require_min_compat_client": "luminous", 2024-09-13T12:11:45.777 INFO:tasks.workunit.client.0.smithi032.stdout: "min_compat_client": "jewel", 2024-09-13T12:11:45.777 INFO:tasks.workunit.client.0.smithi032.stdout: "require_osd_release": "squid", 2024-09-13T12:11:45.777 INFO:tasks.workunit.client.0.smithi032.stdout: "allow_crimson": false, 2024-09-13T12:11:45.777 INFO:tasks.workunit.client.0.smithi032.stdout: "pools": [ 2024-09-13T12:11:45.777 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.778 INFO:tasks.workunit.client.0.smithi032.stdout: "pool": 1, 2024-09-13T12:11:45.778 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_name": "test", 2024-09-13T12:11:45.778 INFO:tasks.workunit.client.0.smithi032.stdout: "create_time": "2024-09-13T12:09:23.852748+0000", 2024-09-13T12:11:45.778 INFO:tasks.workunit.client.0.smithi032.stdout: "flags": 1, 2024-09-13T12:11:45.778 INFO:tasks.workunit.client.0.smithi032.stdout: "flags_names": "hashpspool", 2024-09-13T12:11:45.778 INFO:tasks.workunit.client.0.smithi032.stdout: "type": 1, 2024-09-13T12:11:45.778 INFO:tasks.workunit.client.0.smithi032.stdout: "size": 3, 2024-09-13T12:11:45.778 INFO:tasks.workunit.client.0.smithi032.stdout: "min_size": 2, 2024-09-13T12:11:45.778 INFO:tasks.workunit.client.0.smithi032.stdout: "crush_rule": 0, 2024-09-13T12:11:45.778 INFO:tasks.workunit.client.0.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-13T12:11:45.778 INFO:tasks.workunit.client.0.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-13T12:11:45.778 INFO:tasks.workunit.client.0.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T12:11:45.779 INFO:tasks.workunit.client.0.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T12:11:45.779 INFO:tasks.workunit.client.0.smithi032.stdout: "is_stretch_pool": false, 2024-09-13T12:11:45.779 INFO:tasks.workunit.client.0.smithi032.stdout: "object_hash": 2, 2024-09-13T12:11:45.779 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_autoscale_mode": "on", 2024-09-13T12:11:45.779 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_num": 32, 2024-09-13T12:11:45.779 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_placement_num": 14, 2024-09-13T12:11:45.779 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_placement_num_target": 32, 2024-09-13T12:11:45.779 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_num_target": 32, 2024-09-13T12:11:45.779 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_num_pending": 32, 2024-09-13T12:11:45.779 INFO:tasks.workunit.client.0.smithi032.stdout: "last_pg_merge_meta": { 2024-09-13T12:11:45.779 INFO:tasks.workunit.client.0.smithi032.stdout: "source_pgid": "0.0", 2024-09-13T12:11:45.779 INFO:tasks.workunit.client.0.smithi032.stdout: "ready_epoch": 0, 2024-09-13T12:11:45.779 INFO:tasks.workunit.client.0.smithi032.stdout: "last_epoch_started": 0, 2024-09-13T12:11:45.780 INFO:tasks.workunit.client.0.smithi032.stdout: "last_epoch_clean": 0, 2024-09-13T12:11:45.780 INFO:tasks.workunit.client.0.smithi032.stdout: "source_version": "0'0", 2024-09-13T12:11:45.780 INFO:tasks.workunit.client.0.smithi032.stdout: "target_version": "0'0" 2024-09-13T12:11:45.780 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.780 INFO:tasks.workunit.client.0.smithi032.stdout: "last_change": "55", 2024-09-13T12:11:45.780 INFO:tasks.workunit.client.0.smithi032.stdout: "last_force_op_resend": "0", 2024-09-13T12:11:45.780 INFO:tasks.workunit.client.0.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T12:11:45.780 INFO:tasks.workunit.client.0.smithi032.stdout: "last_force_op_resend_preluminous": "33", 2024-09-13T12:11:45.780 INFO:tasks.workunit.client.0.smithi032.stdout: "auid": 0, 2024-09-13T12:11:45.780 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_mode": "selfmanaged", 2024-09-13T12:11:45.780 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_seq": 0, 2024-09-13T12:11:45.780 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_epoch": 0, 2024-09-13T12:11:45.781 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_snaps": [], 2024-09-13T12:11:45.781 INFO:tasks.workunit.client.0.smithi032.stdout: "removed_snaps": "[]", 2024-09-13T12:11:45.781 INFO:tasks.workunit.client.0.smithi032.stdout: "quota_max_bytes": 0, 2024-09-13T12:11:45.781 INFO:tasks.workunit.client.0.smithi032.stdout: "quota_max_objects": 0, 2024-09-13T12:11:45.781 INFO:tasks.workunit.client.0.smithi032.stdout: "tiers": [], 2024-09-13T12:11:45.781 INFO:tasks.workunit.client.0.smithi032.stdout: "tier_of": -1, 2024-09-13T12:11:45.781 INFO:tasks.workunit.client.0.smithi032.stdout: "read_tier": -1, 2024-09-13T12:11:45.781 INFO:tasks.workunit.client.0.smithi032.stdout: "write_tier": -1, 2024-09-13T12:11:45.781 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_mode": "none", 2024-09-13T12:11:45.781 INFO:tasks.workunit.client.0.smithi032.stdout: "target_max_bytes": 0, 2024-09-13T12:11:45.781 INFO:tasks.workunit.client.0.smithi032.stdout: "target_max_objects": 0, 2024-09-13T12:11:45.782 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T12:11:45.782 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T12:11:45.782 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T12:11:45.782 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-13T12:11:45.782 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-13T12:11:45.782 INFO:tasks.workunit.client.0.smithi032.stdout: "erasure_code_profile": "", 2024-09-13T12:11:45.782 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_params": { 2024-09-13T12:11:45.782 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "none" 2024-09-13T12:11:45.782 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.782 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_period": 0, 2024-09-13T12:11:45.783 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_count": 0, 2024-09-13T12:11:45.783 INFO:tasks.workunit.client.0.smithi032.stdout: "use_gmt_hitset": true, 2024-09-13T12:11:45.783 INFO:tasks.workunit.client.0.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-13T12:11:45.783 INFO:tasks.workunit.client.0.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-13T12:11:45.783 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T12:11:45.783 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-13T12:11:45.783 INFO:tasks.workunit.client.0.smithi032.stdout: "grade_table": [], 2024-09-13T12:11:45.783 INFO:tasks.workunit.client.0.smithi032.stdout: "stripe_width": 0, 2024-09-13T12:11:45.783 INFO:tasks.workunit.client.0.smithi032.stdout: "expected_num_objects": 0, 2024-09-13T12:11:45.783 INFO:tasks.workunit.client.0.smithi032.stdout: "fast_read": false, 2024-09-13T12:11:45.783 INFO:tasks.workunit.client.0.smithi032.stdout: "options": {}, 2024-09-13T12:11:45.783 INFO:tasks.workunit.client.0.smithi032.stdout: "application_metadata": {}, 2024-09-13T12:11:45.783 INFO:tasks.workunit.client.0.smithi032.stdout: "read_balance": { 2024-09-13T12:11:45.784 INFO:tasks.workunit.client.0.smithi032.stdout: "score_type": "Fair distribution", 2024-09-13T12:11:45.784 INFO:tasks.workunit.client.0.smithi032.stdout: "score_acting": 1.690000057220459, 2024-09-13T12:11:45.784 INFO:tasks.workunit.client.0.smithi032.stdout: "score_stable": 1.690000057220459, 2024-09-13T12:11:45.784 INFO:tasks.workunit.client.0.smithi032.stdout: "optimal_score": 0.67000001668930054, 2024-09-13T12:11:45.784 INFO:tasks.workunit.client.0.smithi032.stdout: "raw_score_acting": 1.1299999952316284, 2024-09-13T12:11:45.784 INFO:tasks.workunit.client.0.smithi032.stdout: "raw_score_stable": 1.1299999952316284, 2024-09-13T12:11:45.784 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_affinity_weighted": 1, 2024-09-13T12:11:45.784 INFO:tasks.workunit.client.0.smithi032.stdout: "average_primary_affinity": 1, 2024-09-13T12:11:45.784 INFO:tasks.workunit.client.0.smithi032.stdout: "average_primary_affinity_weighted": 1 2024-09-13T12:11:45.784 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.784 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.784 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:45.784 INFO:tasks.workunit.client.0.smithi032.stdout: "osds": [ 2024-09-13T12:11:45.785 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.785 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 0, 2024-09-13T12:11:45.785 INFO:tasks.workunit.client.0.smithi032.stdout: "uuid": "10a21bfc-e16b-4a73-8068-b8ae59576a72", 2024-09-13T12:11:45.785 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 1, 2024-09-13T12:11:45.785 INFO:tasks.workunit.client.0.smithi032.stdout: "in": 1, 2024-09-13T12:11:45.785 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 1, 2024-09-13T12:11:45.785 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_affinity": 1, 2024-09-13T12:11:45.785 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_begin": 5, 2024-09-13T12:11:45.785 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_end": 50, 2024-09-13T12:11:45.785 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 63, 2024-09-13T12:11:45.786 INFO:tasks.workunit.client.0.smithi032.stdout: "up_thru": 69, 2024-09-13T12:11:45.786 INFO:tasks.workunit.client.0.smithi032.stdout: "down_at": 57, 2024-09-13T12:11:45.786 INFO:tasks.workunit.client.0.smithi032.stdout: "lost_at": 0, 2024-09-13T12:11:45.786 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addrs": { 2024-09-13T12:11:45.786 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:11:45.786 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.786 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:11:45.786 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6802", 2024-09-13T12:11:45.786 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 2302613541 2024-09-13T12:11:45.786 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.786 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.786 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:11:45.787 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6803", 2024-09-13T12:11:45.787 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 2302613541 2024-09-13T12:11:45.787 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.787 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:45.787 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.787 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addrs": { 2024-09-13T12:11:45.787 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:11:45.787 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.787 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:11:45.787 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6804", 2024-09-13T12:11:45.788 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 2302613541 2024-09-13T12:11:45.788 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.788 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.788 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:11:45.788 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6805", 2024-09-13T12:11:45.788 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 2302613541 2024-09-13T12:11:45.788 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.788 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:45.788 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.788 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-13T12:11:45.788 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:11:45.788 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.789 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:11:45.789 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6808", 2024-09-13T12:11:45.789 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 2302613541 2024-09-13T12:11:45.789 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.789 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.789 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:11:45.789 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6809", 2024-09-13T12:11:45.789 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 2302613541 2024-09-13T12:11:45.789 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.789 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:45.789 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.789 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-13T12:11:45.790 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:11:45.790 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.790 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:11:45.790 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6806", 2024-09-13T12:11:45.790 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 2302613541 2024-09-13T12:11:45.790 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.790 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.790 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:11:45.790 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6807", 2024-09-13T12:11:45.790 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 2302613541 2024-09-13T12:11:45.790 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.790 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:45.791 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.791 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addr": "127.0.0.1:6803/2302613541", 2024-09-13T12:11:45.791 INFO:tasks.workunit.client.0.smithi032.stderr:report 2180585037 2024-09-13T12:11:45.792 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addr": "127.0.0.1:6805/2302613541", 2024-09-13T12:11:45.792 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addr": "127.0.0.1:6809/2302613541", 2024-09-13T12:11:45.792 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addr": "127.0.0.1:6807/2302613541", 2024-09-13T12:11:45.792 INFO:tasks.workunit.client.0.smithi032.stdout: "state": [ 2024-09-13T12:11:45.792 INFO:tasks.workunit.client.0.smithi032.stdout: "exists", 2024-09-13T12:11:45.792 INFO:tasks.workunit.client.0.smithi032.stdout: "up" 2024-09-13T12:11:45.793 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:45.793 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.793 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.793 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 1, 2024-09-13T12:11:45.793 INFO:tasks.workunit.client.0.smithi032.stdout: "uuid": "f377d2cc-d182-4459-bd79-b23c9379a371", 2024-09-13T12:11:45.793 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 0, 2024-09-13T12:11:45.793 INFO:tasks.workunit.client.0.smithi032.stdout: "in": 1, 2024-09-13T12:11:45.793 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 1, 2024-09-13T12:11:45.793 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_affinity": 1, 2024-09-13T12:11:45.793 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_begin": 0, 2024-09-13T12:11:45.793 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_end": 0, 2024-09-13T12:11:45.793 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 10, 2024-09-13T12:11:45.794 INFO:tasks.workunit.client.0.smithi032.stdout: "up_thru": 52, 2024-09-13T12:11:45.794 INFO:tasks.workunit.client.0.smithi032.stdout: "down_at": 58, 2024-09-13T12:11:45.794 INFO:tasks.workunit.client.0.smithi032.stdout: "lost_at": 0, 2024-09-13T12:11:45.794 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addrs": { 2024-09-13T12:11:45.794 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:11:45.794 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.794 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:11:45.794 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6810", 2024-09-13T12:11:45.794 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1424248451 2024-09-13T12:11:45.794 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.794 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.794 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:11:45.795 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6811", 2024-09-13T12:11:45.795 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1424248451 2024-09-13T12:11:45.795 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.795 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:45.795 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.795 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addrs": { 2024-09-13T12:11:45.795 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:11:45.795 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.795 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:11:45.795 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6812", 2024-09-13T12:11:45.795 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1424248451 2024-09-13T12:11:45.796 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.796 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.796 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:11:45.796 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6813", 2024-09-13T12:11:45.796 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1424248451 2024-09-13T12:11:45.796 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.796 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:45.796 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.796 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-13T12:11:45.796 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:11:45.796 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.796 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:11:45.797 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6816", 2024-09-13T12:11:45.797 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1424248451 2024-09-13T12:11:45.797 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.797 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.797 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:11:45.797 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6817", 2024-09-13T12:11:45.797 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1424248451 2024-09-13T12:11:45.797 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.797 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:45.797 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.797 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-13T12:11:45.797 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:11:45.798 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.798 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:11:45.798 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6814", 2024-09-13T12:11:45.798 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1424248451 2024-09-13T12:11:45.798 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.798 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.798 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:11:45.798 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6815", 2024-09-13T12:11:45.798 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 1424248451 2024-09-13T12:11:45.798 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.798 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:45.799 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.799 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addr": "127.0.0.1:6811/1424248451", 2024-09-13T12:11:45.799 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addr": "127.0.0.1:6813/1424248451", 2024-09-13T12:11:45.799 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1424248451", 2024-09-13T12:11:45.799 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1424248451", 2024-09-13T12:11:45.799 INFO:tasks.workunit.client.0.smithi032.stdout: "state": [ 2024-09-13T12:11:45.799 INFO:tasks.workunit.client.0.smithi032.stdout: "exists" 2024-09-13T12:11:45.799 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:45.799 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.799 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.799 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2, 2024-09-13T12:11:45.799 INFO:tasks.workunit.client.0.smithi032.stdout: "uuid": "6ac797ab-1924-4ff8-9169-021a47e84482", 2024-09-13T12:11:45.800 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 1, 2024-09-13T12:11:45.800 INFO:tasks.workunit.client.0.smithi032.stdout: "in": 1, 2024-09-13T12:11:45.800 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 1, 2024-09-13T12:11:45.800 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_affinity": 1, 2024-09-13T12:11:45.800 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_begin": 15, 2024-09-13T12:11:45.800 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_end": 51, 2024-09-13T12:11:45.800 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 65, 2024-09-13T12:11:45.800 INFO:tasks.workunit.client.0.smithi032.stdout: "up_thru": 71, 2024-09-13T12:11:45.800 INFO:tasks.workunit.client.0.smithi032.stdout: "down_at": 59, 2024-09-13T12:11:45.800 INFO:tasks.workunit.client.0.smithi032.stdout: "lost_at": 0, 2024-09-13T12:11:45.801 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addrs": { 2024-09-13T12:11:45.801 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:11:45.801 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.801 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:11:45.801 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6810", 2024-09-13T12:11:45.801 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 73367264 2024-09-13T12:11:45.801 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.801 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.801 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:11:45.801 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6811", 2024-09-13T12:11:45.801 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 73367264 2024-09-13T12:11:45.801 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.802 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:45.802 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.802 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addrs": { 2024-09-13T12:11:45.802 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:11:45.802 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.802 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:11:45.802 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6812", 2024-09-13T12:11:45.802 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 73367264 2024-09-13T12:11:45.802 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.802 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.803 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:11:45.803 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6813", 2024-09-13T12:11:45.803 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 73367264 2024-09-13T12:11:45.803 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.803 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:45.803 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.803 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-13T12:11:45.803 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:11:45.803 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.803 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:11:45.803 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6816", 2024-09-13T12:11:45.803 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 73367264 2024-09-13T12:11:45.804 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.804 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.804 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:11:45.804 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6817", 2024-09-13T12:11:45.804 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 73367264 2024-09-13T12:11:45.804 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.804 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:45.804 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.804 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-13T12:11:45.804 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:11:45.804 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.804 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:11:45.805 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6814", 2024-09-13T12:11:45.805 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 73367264 2024-09-13T12:11:45.805 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.805 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.805 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:11:45.805 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6815", 2024-09-13T12:11:45.805 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 73367264 2024-09-13T12:11:45.805 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.805 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:45.805 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.805 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addr": "127.0.0.1:6811/73367264", 2024-09-13T12:11:45.805 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addr": "127.0.0.1:6813/73367264", 2024-09-13T12:11:45.806 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addr": "127.0.0.1:6817/73367264", 2024-09-13T12:11:45.806 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addr": "127.0.0.1:6815/73367264", 2024-09-13T12:11:45.806 INFO:tasks.workunit.client.0.smithi032.stdout: "state": [ 2024-09-13T12:11:45.806 INFO:tasks.workunit.client.0.smithi032.stdout: "exists", 2024-09-13T12:11:45.806 INFO:tasks.workunit.client.0.smithi032.stdout: "up" 2024-09-13T12:11:45.806 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:45.806 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.806 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:45.806 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_xinfo": [ 2024-09-13T12:11:45.806 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.806 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 0, 2024-09-13T12:11:45.807 INFO:tasks.workunit.client.0.smithi032.stdout: "down_stamp": "2024-09-13T12:10:16.065537+0000", 2024-09-13T12:11:45.807 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_probability": 0, 2024-09-13T12:11:45.807 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_interval": 0, 2024-09-13T12:11:45.807 INFO:tasks.workunit.client.0.smithi032.stdout: "features": 4540719139924082687, 2024-09-13T12:11:45.807 INFO:tasks.workunit.client.0.smithi032.stdout: "old_weight": 0, 2024-09-13T12:11:45.807 INFO:tasks.workunit.client.0.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-13T12:08:09.703357+0000", 2024-09-13T12:11:45.807 INFO:tasks.workunit.client.0.smithi032.stdout: "dead_epoch": 0 2024-09-13T12:11:45.807 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.807 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.807 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 1, 2024-09-13T12:11:45.807 INFO:tasks.workunit.client.0.smithi032.stdout: "down_stamp": "2024-09-13T12:10:16.671856+0000", 2024-09-13T12:11:45.807 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_probability": 0, 2024-09-13T12:11:45.808 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_interval": 0, 2024-09-13T12:11:45.808 INFO:tasks.workunit.client.0.smithi032.stdout: "features": 4540719139924082687, 2024-09-13T12:11:45.808 INFO:tasks.workunit.client.0.smithi032.stdout: "old_weight": 0, 2024-09-13T12:11:45.808 INFO:tasks.workunit.client.0.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-13T12:08:44.286568+0000", 2024-09-13T12:11:45.808 INFO:tasks.workunit.client.0.smithi032.stdout: "dead_epoch": 0 2024-09-13T12:11:45.808 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.808 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.808 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2, 2024-09-13T12:11:45.808 INFO:tasks.workunit.client.0.smithi032.stdout: "down_stamp": "2024-09-13T12:10:17.200480+0000", 2024-09-13T12:11:45.808 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_probability": 0, 2024-09-13T12:11:45.808 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_interval": 0, 2024-09-13T12:11:45.809 INFO:tasks.workunit.client.0.smithi032.stdout: "features": 4540719139924082687, 2024-09-13T12:11:45.809 INFO:tasks.workunit.client.0.smithi032.stdout: "old_weight": 0, 2024-09-13T12:11:45.809 INFO:tasks.workunit.client.0.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-13T12:09:20.289467+0000", 2024-09-13T12:11:45.809 INFO:tasks.workunit.client.0.smithi032.stdout: "dead_epoch": 0 2024-09-13T12:11:45.809 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.809 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:45.809 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_upmap": [], 2024-09-13T12:11:45.809 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_upmap_items": [], 2024-09-13T12:11:45.809 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_upmap_primaries": [], 2024-09-13T12:11:45.809 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_temp": [], 2024-09-13T12:11:45.809 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_temp": [], 2024-09-13T12:11:45.810 INFO:tasks.workunit.client.0.smithi032.stdout: "blocklist": {}, 2024-09-13T12:11:45.810 INFO:tasks.workunit.client.0.smithi032.stdout: "range_blocklist": {}, 2024-09-13T12:11:45.810 INFO:tasks.workunit.client.0.smithi032.stdout: "erasure_code_profiles": { 2024-09-13T12:11:45.810 INFO:tasks.workunit.client.0.smithi032.stdout: "default": { 2024-09-13T12:11:45.810 INFO:tasks.workunit.client.0.smithi032.stdout: "k": "2", 2024-09-13T12:11:45.810 INFO:tasks.workunit.client.0.smithi032.stdout: "m": "2", 2024-09-13T12:11:45.810 INFO:tasks.workunit.client.0.smithi032.stdout: "plugin": "jerasure", 2024-09-13T12:11:45.810 INFO:tasks.workunit.client.0.smithi032.stdout: "technique": "reed_sol_van" 2024-09-13T12:11:45.810 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.810 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.810 INFO:tasks.workunit.client.0.smithi032.stdout: "removed_snaps_queue": [], 2024-09-13T12:11:45.811 INFO:tasks.workunit.client.0.smithi032.stdout: "new_removed_snaps": [], 2024-09-13T12:11:45.811 INFO:tasks.workunit.client.0.smithi032.stdout: "new_purged_snaps": [], 2024-09-13T12:11:45.811 INFO:tasks.workunit.client.0.smithi032.stdout: "crush_node_flags": {}, 2024-09-13T12:11:45.811 INFO:tasks.workunit.client.0.smithi032.stdout: "device_class_flags": {}, 2024-09-13T12:11:45.811 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_mode": { 2024-09-13T12:11:45.811 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_mode_enabled": false, 2024-09-13T12:11:45.811 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_bucket_count": 0, 2024-09-13T12:11:45.811 INFO:tasks.workunit.client.0.smithi032.stdout: "degraded_stretch_mode": 0, 2024-09-13T12:11:45.811 INFO:tasks.workunit.client.0.smithi032.stdout: "recovering_stretch_mode": 0, 2024-09-13T12:11:45.811 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_mode_bucket": 0 2024-09-13T12:11:45.811 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.811 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.811 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_metadata": [ 2024-09-13T12:11:45.812 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.812 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:11:45.812 INFO:tasks.workunit.client.0.smithi032.stdout: "arch": "x86_64", 2024-09-13T12:11:45.812 INFO:tasks.workunit.client.0.smithi032.stdout: "back_addr": "[v2:127.0.0.1:6804/2302613541,v1:127.0.0.1:6805/2302613541]", 2024-09-13T12:11:45.812 INFO:tasks.workunit.client.0.smithi032.stdout: "back_iface": "", 2024-09-13T12:11:45.812 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs": "1", 2024-09-13T12:11:45.812 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_db": "0", 2024-09-13T12:11:45.812 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_wal": "0", 2024-09-13T12:11:45.812 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_single_shared_device": "1", 2024-09-13T12:11:45.812 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_allocation_from_file": "0", 2024-09-13T12:11:45.813 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_access_mode": "file", 2024-09-13T12:11:45.813 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_block_size": "4096", 2024-09-13T12:11:45.813 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_devices": "sda", 2024-09-13T12:11:45.813 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-13T12:11:45.813 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-13T12:11:45.813 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-13T12:11:45.813 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_rotational": "1", 2024-09-13T12:11:45.813 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_size": "107374182400", 2024-09-13T12:11:45.813 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_support_discard": "1", 2024-09-13T12:11:45.813 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_type": "hdd", 2024-09-13T12:11:45.813 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_min_alloc_size": "4096", 2024-09-13T12:11:45.813 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_release": "squid", 2024-09-13T12:11:45.814 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:11:45.814 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_short": "19.3.0-4697-gd58f7306", 2024-09-13T12:11:45.814 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_when_created": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:11:45.814 INFO:tasks.workunit.client.0.smithi032.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-13T12:11:45.814 INFO:tasks.workunit.client.0.smithi032.stdout: "created_at": "2024-09-13T12:07:41.934841Z", 2024-09-13T12:11:45.814 INFO:tasks.workunit.client.0.smithi032.stdout: "default_device_class": "hdd", 2024-09-13T12:11:45.814 INFO:tasks.workunit.client.0.smithi032.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4J2LL", 2024-09-13T12:11:45.814 INFO:tasks.workunit.client.0.smithi032.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-13T12:11:45.814 INFO:tasks.workunit.client.0.smithi032.stdout: "devices": "sda", 2024-09-13T12:11:45.814 INFO:tasks.workunit.client.0.smithi032.stdout: "distro": "centos", 2024-09-13T12:11:45.814 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_description": "CentOS Stream 9", 2024-09-13T12:11:45.814 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_version": "9", 2024-09-13T12:11:45.814 INFO:tasks.workunit.client.0.smithi032.stdout: "front_addr": "[v2:127.0.0.1:6802/2302613541,v1:127.0.0.1:6803/2302613541]", 2024-09-13T12:11:45.815 INFO:tasks.workunit.client.0.smithi032.stdout: "front_iface": "", 2024-09-13T12:11:45.815 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/2302613541,v1:127.0.0.1:6809/2302613541]", 2024-09-13T12:11:45.815 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/2302613541,v1:127.0.0.1:6807/2302613541]", 2024-09-13T12:11:45.815 INFO:tasks.workunit.client.0.smithi032.stdout: "hostname": "smithi032", 2024-09-13T12:11:45.815 INFO:tasks.workunit.client.0.smithi032.stdout: "journal_rotational": "1", 2024-09-13T12:11:45.815 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-13T12:11:45.815 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-13T12:11:45.815 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_swap_kb": "0", 2024-09-13T12:11:45.815 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_total_kb": "32493484", 2024-09-13T12:11:45.815 INFO:tasks.workunit.client.0.smithi032.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-13T12:11:45.815 INFO:tasks.workunit.client.0.smithi032.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-13T12:11:45.816 INFO:tasks.workunit.client.0.smithi032.stdout: "os": "Linux", 2024-09-13T12:11:45.816 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_data": "td/divergent-priors/0", 2024-09-13T12:11:45.816 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_objectstore": "bluestore", 2024-09-13T12:11:45.816 INFO:tasks.workunit.client.0.smithi032.stdout: "osdspec_affinity": "", 2024-09-13T12:11:45.816 INFO:tasks.workunit.client.0.smithi032.stdout: "rotational": "1" 2024-09-13T12:11:45.816 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.816 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.816 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 1, 2024-09-13T12:11:45.816 INFO:tasks.workunit.client.0.smithi032.stdout: "arch": "x86_64", 2024-09-13T12:11:45.816 INFO:tasks.workunit.client.0.smithi032.stdout: "back_addr": "[v2:127.0.0.1:6812/1424248451,v1:127.0.0.1:6813/1424248451]", 2024-09-13T12:11:45.816 INFO:tasks.workunit.client.0.smithi032.stdout: "back_iface": "", 2024-09-13T12:11:45.817 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs": "1", 2024-09-13T12:11:45.817 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_db": "0", 2024-09-13T12:11:45.817 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_wal": "0", 2024-09-13T12:11:45.817 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_single_shared_device": "1", 2024-09-13T12:11:45.817 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_allocation_from_file": "0", 2024-09-13T12:11:45.817 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_access_mode": "file", 2024-09-13T12:11:45.817 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_block_size": "4096", 2024-09-13T12:11:45.817 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_devices": "sda", 2024-09-13T12:11:45.817 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-13T12:11:45.817 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-13T12:11:45.817 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-13T12:11:45.817 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_rotational": "1", 2024-09-13T12:11:45.818 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_size": "107374182400", 2024-09-13T12:11:45.818 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_support_discard": "1", 2024-09-13T12:11:45.818 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_type": "hdd", 2024-09-13T12:11:45.818 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_min_alloc_size": "4096", 2024-09-13T12:11:45.818 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_release": "squid", 2024-09-13T12:11:45.818 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:11:45.818 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_short": "19.3.0-4697-gd58f7306", 2024-09-13T12:11:45.818 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_when_created": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:11:45.818 INFO:tasks.workunit.client.0.smithi032.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-13T12:11:45.818 INFO:tasks.workunit.client.0.smithi032.stdout: "created_at": "2024-09-13T12:08:16.197013Z", 2024-09-13T12:11:45.818 INFO:tasks.workunit.client.0.smithi032.stdout: "default_device_class": "hdd", 2024-09-13T12:11:45.818 INFO:tasks.workunit.client.0.smithi032.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4J2LL", 2024-09-13T12:11:45.818 INFO:tasks.workunit.client.0.smithi032.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-13T12:11:45.819 INFO:tasks.workunit.client.0.smithi032.stdout: "devices": "sda", 2024-09-13T12:11:45.819 INFO:tasks.workunit.client.0.smithi032.stdout: "distro": "centos", 2024-09-13T12:11:45.819 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_description": "CentOS Stream 9", 2024-09-13T12:11:45.819 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_version": "9", 2024-09-13T12:11:45.819 INFO:tasks.workunit.client.0.smithi032.stdout: "front_addr": "[v2:127.0.0.1:6810/1424248451,v1:127.0.0.1:6811/1424248451]", 2024-09-13T12:11:45.819 INFO:tasks.workunit.client.0.smithi032.stdout: "front_iface": "", 2024-09-13T12:11:45.819 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1424248451,v1:127.0.0.1:6817/1424248451]", 2024-09-13T12:11:45.819 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1424248451,v1:127.0.0.1:6815/1424248451]", 2024-09-13T12:11:45.819 INFO:tasks.workunit.client.0.smithi032.stdout: "hostname": "smithi032", 2024-09-13T12:11:45.819 INFO:tasks.workunit.client.0.smithi032.stdout: "journal_rotational": "1", 2024-09-13T12:11:45.819 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-13T12:11:45.820 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-13T12:11:45.820 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_swap_kb": "0", 2024-09-13T12:11:45.820 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_total_kb": "32493484", 2024-09-13T12:11:45.820 INFO:tasks.workunit.client.0.smithi032.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-13T12:11:45.820 INFO:tasks.workunit.client.0.smithi032.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-13T12:11:45.820 INFO:tasks.workunit.client.0.smithi032.stdout: "os": "Linux", 2024-09-13T12:11:45.820 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_data": "td/divergent-priors/1", 2024-09-13T12:11:45.820 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_objectstore": "bluestore", 2024-09-13T12:11:45.820 INFO:tasks.workunit.client.0.smithi032.stdout: "osdspec_affinity": "", 2024-09-13T12:11:45.820 INFO:tasks.workunit.client.0.smithi032.stdout: "rotational": "1" 2024-09-13T12:11:45.820 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.820 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.821 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:11:45.821 INFO:tasks.workunit.client.0.smithi032.stdout: "arch": "x86_64", 2024-09-13T12:11:45.821 INFO:tasks.workunit.client.0.smithi032.stdout: "back_addr": "[v2:127.0.0.1:6812/73367264,v1:127.0.0.1:6813/73367264]", 2024-09-13T12:11:45.821 INFO:tasks.workunit.client.0.smithi032.stdout: "back_iface": "", 2024-09-13T12:11:45.821 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs": "1", 2024-09-13T12:11:45.821 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_db": "0", 2024-09-13T12:11:45.821 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_wal": "0", 2024-09-13T12:11:45.821 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_single_shared_device": "1", 2024-09-13T12:11:45.821 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_allocation_from_file": "0", 2024-09-13T12:11:45.821 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_access_mode": "file", 2024-09-13T12:11:45.821 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_block_size": "4096", 2024-09-13T12:11:45.822 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_devices": "sda", 2024-09-13T12:11:45.822 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-13T12:11:45.822 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-13T12:11:45.822 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-13T12:11:45.822 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_rotational": "1", 2024-09-13T12:11:45.822 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_size": "107374182400", 2024-09-13T12:11:45.822 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_support_discard": "1", 2024-09-13T12:11:45.822 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_type": "hdd", 2024-09-13T12:11:45.822 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_min_alloc_size": "4096", 2024-09-13T12:11:45.822 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_release": "squid", 2024-09-13T12:11:45.822 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:11:45.823 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_short": "19.3.0-4697-gd58f7306", 2024-09-13T12:11:45.823 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_when_created": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:11:45.823 INFO:tasks.workunit.client.0.smithi032.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-13T12:11:45.823 INFO:tasks.workunit.client.0.smithi032.stdout: "created_at": "2024-09-13T12:08:52.300025Z", 2024-09-13T12:11:45.823 INFO:tasks.workunit.client.0.smithi032.stdout: "default_device_class": "hdd", 2024-09-13T12:11:45.823 INFO:tasks.workunit.client.0.smithi032.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4J2LL", 2024-09-13T12:11:45.823 INFO:tasks.workunit.client.0.smithi032.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-13T12:11:45.823 INFO:tasks.workunit.client.0.smithi032.stdout: "devices": "sda", 2024-09-13T12:11:45.823 INFO:tasks.workunit.client.0.smithi032.stdout: "distro": "centos", 2024-09-13T12:11:45.823 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_description": "CentOS Stream 9", 2024-09-13T12:11:45.823 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_version": "9", 2024-09-13T12:11:45.824 INFO:tasks.workunit.client.0.smithi032.stdout: "front_addr": "[v2:127.0.0.1:6810/73367264,v1:127.0.0.1:6811/73367264]", 2024-09-13T12:11:45.824 INFO:tasks.workunit.client.0.smithi032.stdout: "front_iface": "", 2024-09-13T12:11:45.824 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/73367264,v1:127.0.0.1:6817/73367264]", 2024-09-13T12:11:45.824 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/73367264,v1:127.0.0.1:6815/73367264]", 2024-09-13T12:11:45.824 INFO:tasks.workunit.client.0.smithi032.stdout: "hostname": "smithi032", 2024-09-13T12:11:45.824 INFO:tasks.workunit.client.0.smithi032.stdout: "journal_rotational": "1", 2024-09-13T12:11:45.824 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-13T12:11:45.824 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-13T12:11:45.824 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_swap_kb": "0", 2024-09-13T12:11:45.825 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_total_kb": "32493484", 2024-09-13T12:11:45.825 INFO:tasks.workunit.client.0.smithi032.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-13T12:11:45.825 INFO:tasks.workunit.client.0.smithi032.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-13T12:11:45.825 INFO:tasks.workunit.client.0.smithi032.stdout: "os": "Linux", 2024-09-13T12:11:45.825 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_data": "td/divergent-priors/2", 2024-09-13T12:11:45.825 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_objectstore": "bluestore", 2024-09-13T12:11:45.825 INFO:tasks.workunit.client.0.smithi032.stdout: "osdspec_affinity": "", 2024-09-13T12:11:45.825 INFO:tasks.workunit.client.0.smithi032.stdout: "rotational": "1" 2024-09-13T12:11:45.825 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.825 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:45.825 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap_clean_epochs": { 2024-09-13T12:11:45.826 INFO:tasks.workunit.client.0.smithi032.stdout: "min_last_epoch_clean": 0, 2024-09-13T12:11:45.826 INFO:tasks.workunit.client.0.smithi032.stdout: "last_epoch_clean": { 2024-09-13T12:11:45.826 INFO:tasks.workunit.client.0.smithi032.stdout: "per_pool": [ 2024-09-13T12:11:45.826 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.826 INFO:tasks.workunit.client.0.smithi032.stdout: "poolid": 1, 2024-09-13T12:11:45.826 INFO:tasks.workunit.client.0.smithi032.stdout: "floor": 36 2024-09-13T12:11:45.826 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.826 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:45.826 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.827 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_epochs": [ 2024-09-13T12:11:45.827 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.827 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:11:45.827 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 63 2024-09-13T12:11:45.827 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.827 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.827 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:11:45.827 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 65 2024-09-13T12:11:45.827 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.827 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:45.828 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.828 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap_first_committed": 1, 2024-09-13T12:11:45.828 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap_last_committed": 73, 2024-09-13T12:11:45.828 INFO:tasks.workunit.client.0.smithi032.stdout: "crushmap": { 2024-09-13T12:11:45.828 INFO:tasks.workunit.client.0.smithi032.stdout: "devices": [ 2024-09-13T12:11:45.828 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.828 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:11:45.828 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "osd.0", 2024-09-13T12:11:45.828 INFO:tasks.workunit.client.0.smithi032.stdout: "class": "hdd" 2024-09-13T12:11:45.828 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.829 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.829 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 1, 2024-09-13T12:11:45.829 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "osd.1", 2024-09-13T12:11:45.829 INFO:tasks.workunit.client.0.smithi032.stdout: "class": "hdd" 2024-09-13T12:11:45.829 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.829 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.829 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:11:45.829 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "osd.2", 2024-09-13T12:11:45.829 INFO:tasks.workunit.client.0.smithi032.stdout: "class": "hdd" 2024-09-13T12:11:45.829 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.829 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:45.830 INFO:tasks.workunit.client.0.smithi032.stdout: "types": [ 2024-09-13T12:11:45.830 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.830 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 0, 2024-09-13T12:11:45.830 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "osd" 2024-09-13T12:11:45.830 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.830 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.830 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 1, 2024-09-13T12:11:45.830 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "host" 2024-09-13T12:11:45.831 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.831 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.831 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 2, 2024-09-13T12:11:45.831 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "chassis" 2024-09-13T12:11:45.831 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.831 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.831 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 3, 2024-09-13T12:11:45.831 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "rack" 2024-09-13T12:11:45.831 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.831 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.831 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 4, 2024-09-13T12:11:45.832 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "row" 2024-09-13T12:11:45.832 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.832 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.832 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 5, 2024-09-13T12:11:45.832 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "pdu" 2024-09-13T12:11:45.832 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.832 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.832 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 6, 2024-09-13T12:11:45.832 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "pod" 2024-09-13T12:11:45.833 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.833 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.833 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 7, 2024-09-13T12:11:45.833 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "room" 2024-09-13T12:11:45.833 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.833 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.833 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 8, 2024-09-13T12:11:45.833 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "datacenter" 2024-09-13T12:11:45.833 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.833 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.833 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 9, 2024-09-13T12:11:45.834 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "zone" 2024-09-13T12:11:45.834 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.834 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.834 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 10, 2024-09-13T12:11:45.834 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "region" 2024-09-13T12:11:45.834 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.834 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.834 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 11, 2024-09-13T12:11:45.834 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "root" 2024-09-13T12:11:45.834 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.834 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:45.834 INFO:tasks.workunit.client.0.smithi032.stdout: "buckets": [ 2024-09-13T12:11:45.835 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.835 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -1, 2024-09-13T12:11:45.835 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "default", 2024-09-13T12:11:45.835 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 11, 2024-09-13T12:11:45.835 INFO:tasks.workunit.client.0.smithi032.stdout: "type_name": "root", 2024-09-13T12:11:45.835 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:11:45.835 INFO:tasks.workunit.client.0.smithi032.stdout: "alg": "straw2", 2024-09-13T12:11:45.835 INFO:tasks.workunit.client.0.smithi032.stdout: "hash": "rjenkins1", 2024-09-13T12:11:45.835 INFO:tasks.workunit.client.0.smithi032.stdout: "items": [ 2024-09-13T12:11:45.835 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.835 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -3, 2024-09-13T12:11:45.835 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:11:45.836 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 0 2024-09-13T12:11:45.836 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.836 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:45.836 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.836 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.836 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -2, 2024-09-13T12:11:45.836 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "default~hdd", 2024-09-13T12:11:45.836 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 11, 2024-09-13T12:11:45.836 INFO:tasks.workunit.client.0.smithi032.stdout: "type_name": "root", 2024-09-13T12:11:45.836 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:11:45.836 INFO:tasks.workunit.client.0.smithi032.stdout: "alg": "straw2", 2024-09-13T12:11:45.837 INFO:tasks.workunit.client.0.smithi032.stdout: "hash": "rjenkins1", 2024-09-13T12:11:45.837 INFO:tasks.workunit.client.0.smithi032.stdout: "items": [ 2024-09-13T12:11:45.837 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.837 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -4, 2024-09-13T12:11:45.837 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:11:45.837 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 0 2024-09-13T12:11:45.837 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.837 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:45.837 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.837 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.837 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -3, 2024-09-13T12:11:45.837 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "smithi032", 2024-09-13T12:11:45.838 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 1, 2024-09-13T12:11:45.838 INFO:tasks.workunit.client.0.smithi032.stdout: "type_name": "host", 2024-09-13T12:11:45.838 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:11:45.838 INFO:tasks.workunit.client.0.smithi032.stdout: "alg": "straw2", 2024-09-13T12:11:45.838 INFO:tasks.workunit.client.0.smithi032.stdout: "hash": "rjenkins1", 2024-09-13T12:11:45.838 INFO:tasks.workunit.client.0.smithi032.stdout: "items": [ 2024-09-13T12:11:45.838 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.838 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:11:45.838 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:11:45.838 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 0 2024-09-13T12:11:45.838 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.838 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.839 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 1, 2024-09-13T12:11:45.839 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:11:45.839 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 1 2024-09-13T12:11:45.839 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.839 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.839 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:11:45.839 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:11:45.839 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 2 2024-09-13T12:11:45.839 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.839 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:45.839 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.840 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.840 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -4, 2024-09-13T12:11:45.840 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "smithi032~hdd", 2024-09-13T12:11:45.840 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 1, 2024-09-13T12:11:45.840 INFO:tasks.workunit.client.0.smithi032.stdout: "type_name": "host", 2024-09-13T12:11:45.840 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:11:45.840 INFO:tasks.workunit.client.0.smithi032.stdout: "alg": "straw2", 2024-09-13T12:11:45.840 INFO:tasks.workunit.client.0.smithi032.stdout: "hash": "rjenkins1", 2024-09-13T12:11:45.840 INFO:tasks.workunit.client.0.smithi032.stdout: "items": [ 2024-09-13T12:11:45.840 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.840 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:11:45.840 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:11:45.841 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 0 2024-09-13T12:11:45.841 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.841 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.841 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 1, 2024-09-13T12:11:45.841 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:11:45.841 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 1 2024-09-13T12:11:45.841 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.841 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.841 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:11:45.841 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:11:45.841 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 2 2024-09-13T12:11:45.841 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.842 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:45.842 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.842 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:45.842 INFO:tasks.workunit.client.0.smithi032.stdout: "rules": [ 2024-09-13T12:11:45.842 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.842 INFO:tasks.workunit.client.0.smithi032.stdout: "rule_id": 0, 2024-09-13T12:11:45.842 INFO:tasks.workunit.client.0.smithi032.stdout: "rule_name": "replicated_rule", 2024-09-13T12:11:45.842 INFO:tasks.workunit.client.0.smithi032.stdout: "type": 1, 2024-09-13T12:11:45.842 INFO:tasks.workunit.client.0.smithi032.stdout: "steps": [ 2024-09-13T12:11:45.842 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.842 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "take", 2024-09-13T12:11:45.842 INFO:tasks.workunit.client.0.smithi032.stdout: "item": -1, 2024-09-13T12:11:45.842 INFO:tasks.workunit.client.0.smithi032.stdout: "item_name": "default" 2024-09-13T12:11:45.843 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.843 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.843 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "choose_firstn", 2024-09-13T12:11:45.843 INFO:tasks.workunit.client.0.smithi032.stdout: "num": 0, 2024-09-13T12:11:45.843 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "osd" 2024-09-13T12:11:45.843 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.843 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.843 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "emit" 2024-09-13T12:11:45.843 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.843 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:11:45.843 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.844 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:45.844 INFO:tasks.workunit.client.0.smithi032.stdout: "tunables": { 2024-09-13T12:11:45.844 INFO:tasks.workunit.client.0.smithi032.stdout: "choose_local_tries": 0, 2024-09-13T12:11:45.844 INFO:tasks.workunit.client.0.smithi032.stdout: "choose_local_fallback_tries": 0, 2024-09-13T12:11:45.844 INFO:tasks.workunit.client.0.smithi032.stdout: "choose_total_tries": 50, 2024-09-13T12:11:45.844 INFO:tasks.workunit.client.0.smithi032.stdout: "chooseleaf_descend_once": 1, 2024-09-13T12:11:45.844 INFO:tasks.workunit.client.0.smithi032.stdout: "chooseleaf_vary_r": 1, 2024-09-13T12:11:45.844 INFO:tasks.workunit.client.0.smithi032.stdout: "chooseleaf_stable": 1, 2024-09-13T12:11:45.844 INFO:tasks.workunit.client.0.smithi032.stdout: "msr_descents": 100, 2024-09-13T12:11:45.844 INFO:tasks.workunit.client.0.smithi032.stdout: "msr_collision_tries": 100, 2024-09-13T12:11:45.844 INFO:tasks.workunit.client.0.smithi032.stdout: "straw_calc_version": 1, 2024-09-13T12:11:45.844 INFO:tasks.workunit.client.0.smithi032.stdout: "allowed_bucket_algs": 54, 2024-09-13T12:11:45.845 INFO:tasks.workunit.client.0.smithi032.stdout: "profile": "jewel", 2024-09-13T12:11:45.845 INFO:tasks.workunit.client.0.smithi032.stdout: "optimal_tunables": 1, 2024-09-13T12:11:45.845 INFO:tasks.workunit.client.0.smithi032.stdout: "legacy_tunables": 0, 2024-09-13T12:11:45.845 INFO:tasks.workunit.client.0.smithi032.stdout: "minimum_required_version": "jewel", 2024-09-13T12:11:45.845 INFO:tasks.workunit.client.0.smithi032.stdout: "require_feature_tunables": 1, 2024-09-13T12:11:45.845 INFO:tasks.workunit.client.0.smithi032.stdout: "require_feature_tunables2": 1, 2024-09-13T12:11:45.845 INFO:tasks.workunit.client.0.smithi032.stdout: "has_v2_rules": 0, 2024-09-13T12:11:45.845 INFO:tasks.workunit.client.0.smithi032.stdout: "require_feature_tunables3": 1, 2024-09-13T12:11:45.845 INFO:tasks.workunit.client.0.smithi032.stdout: "has_v3_rules": 0, 2024-09-13T12:11:45.845 INFO:tasks.workunit.client.0.smithi032.stdout: "has_v4_buckets": 1, 2024-09-13T12:11:45.845 INFO:tasks.workunit.client.0.smithi032.stdout: "require_feature_tunables5": 1, 2024-09-13T12:11:45.845 INFO:tasks.workunit.client.0.smithi032.stdout: "has_v5_rules": 0, 2024-09-13T12:11:45.846 INFO:tasks.workunit.client.0.smithi032.stdout: "has_msr_rules": 0 2024-09-13T12:11:45.846 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.846 INFO:tasks.workunit.client.0.smithi032.stdout: "choose_args": {} 2024-09-13T12:11:45.846 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.846 INFO:tasks.workunit.client.0.smithi032.stdout: "fsmap": { 2024-09-13T12:11:45.846 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 1, 2024-09-13T12:11:45.846 INFO:tasks.workunit.client.0.smithi032.stdout: "btime": "2024-09-13T12:07:35:378184+0000", 2024-09-13T12:11:45.846 INFO:tasks.workunit.client.0.smithi032.stdout: "default_fscid": -1, 2024-09-13T12:11:45.846 INFO:tasks.workunit.client.0.smithi032.stdout: "compat": { 2024-09-13T12:11:45.846 INFO:tasks.workunit.client.0.smithi032.stdout: "compat": {}, 2024-09-13T12:11:45.846 INFO:tasks.workunit.client.0.smithi032.stdout: "ro_compat": {}, 2024-09-13T12:11:45.847 INFO:tasks.workunit.client.0.smithi032.stdout: "incompat": { 2024-09-13T12:11:45.847 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_1": "base v0.20", 2024-09-13T12:11:45.847 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_2": "client writeable ranges", 2024-09-13T12:11:45.847 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_3": "default file layouts on dirs", 2024-09-13T12:11:45.847 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_4": "dir inode in separate object", 2024-09-13T12:11:45.847 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_5": "mds uses versioned encoding", 2024-09-13T12:11:45.847 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-13T12:11:45.847 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_8": "no anchor table", 2024-09-13T12:11:45.847 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_9": "file layout v2", 2024-09-13T12:11:45.847 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_10": "snaprealm v2", 2024-09-13T12:11:45.847 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_11": "minor log segments", 2024-09-13T12:11:45.848 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_12": "quiesce subvolumes" 2024-09-13T12:11:45.848 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.848 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.848 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_flags": { 2024-09-13T12:11:45.848 INFO:tasks.workunit.client.0.smithi032.stdout: "enable_multiple": true, 2024-09-13T12:11:45.848 INFO:tasks.workunit.client.0.smithi032.stdout: "ever_enabled_multiple": true 2024-09-13T12:11:45.848 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.848 INFO:tasks.workunit.client.0.smithi032.stdout: "standbys": [], 2024-09-13T12:11:45.848 INFO:tasks.workunit.client.0.smithi032.stdout: "filesystems": [] 2024-09-13T12:11:45.848 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.848 INFO:tasks.workunit.client.0.smithi032.stdout: "mdsmap_first_committed": 1, 2024-09-13T12:11:45.848 INFO:tasks.workunit.client.0.smithi032.stdout: "mdsmap_last_committed": 1, 2024-09-13T12:11:45.849 INFO:tasks.workunit.client.0.smithi032.stdout: "auth": { 2024-09-13T12:11:45.849 INFO:tasks.workunit.client.0.smithi032.stdout: "first_committed": 1, 2024-09-13T12:11:45.849 INFO:tasks.workunit.client.0.smithi032.stdout: "last_committed": 5, 2024-09-13T12:11:45.849 INFO:tasks.workunit.client.0.smithi032.stdout: "num_secrets": 3 2024-09-13T12:11:45.849 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.849 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg": 32, 2024-09-13T12:11:45.849 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg_active": 32, 2024-09-13T12:11:45.849 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg_unknown": 0, 2024-09-13T12:11:45.849 INFO:tasks.workunit.client.0.smithi032.stdout: "num_osd": 3, 2024-09-13T12:11:45.849 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_sum": { 2024-09-13T12:11:45.849 INFO:tasks.workunit.client.0.smithi032.stdout: "stat_sum": { 2024-09-13T12:11:45.849 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes": 48514, 2024-09-13T12:11:45.850 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects": 103, 2024-09-13T12:11:45.850 INFO:tasks.workunit.client.0.smithi032.stdout: "num_object_clones": 0, 2024-09-13T12:11:45.850 INFO:tasks.workunit.client.0.smithi032.stdout: "num_object_copies": 309, 2024-09-13T12:11:45.850 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_missing_on_primary": 0, 2024-09-13T12:11:45.850 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_missing": 0, 2024-09-13T12:11:45.850 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_degraded": 103, 2024-09-13T12:11:45.850 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_misplaced": 0, 2024-09-13T12:11:45.850 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_unfound": 0, 2024-09-13T12:11:45.850 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_dirty": 103, 2024-09-13T12:11:45.850 INFO:tasks.workunit.client.0.smithi032.stdout: "num_whiteouts": 0, 2024-09-13T12:11:45.850 INFO:tasks.workunit.client.0.smithi032.stdout: "num_read": 0, 2024-09-13T12:11:45.851 INFO:tasks.workunit.client.0.smithi032.stdout: "num_read_kb": 0, 2024-09-13T12:11:45.851 INFO:tasks.workunit.client.0.smithi032.stdout: "num_write": 58, 2024-09-13T12:11:45.851 INFO:tasks.workunit.client.0.smithi032.stdout: "num_write_kb": 58, 2024-09-13T12:11:45.851 INFO:tasks.workunit.client.0.smithi032.stdout: "num_scrub_errors": 0, 2024-09-13T12:11:45.851 INFO:tasks.workunit.client.0.smithi032.stdout: "num_shallow_scrub_errors": 0, 2024-09-13T12:11:45.851 INFO:tasks.workunit.client.0.smithi032.stdout: "num_deep_scrub_errors": 0, 2024-09-13T12:11:45.851 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_recovered": 0, 2024-09-13T12:11:45.851 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes_recovered": 0, 2024-09-13T12:11:45.851 INFO:tasks.workunit.client.0.smithi032.stdout: "num_keys_recovered": 0, 2024-09-13T12:11:45.851 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_omap": 0, 2024-09-13T12:11:45.851 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_hit_set_archive": 0, 2024-09-13T12:11:45.851 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes_hit_set_archive": 0, 2024-09-13T12:11:45.851 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush": 0, 2024-09-13T12:11:45.852 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_kb": 0, 2024-09-13T12:11:45.852 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict": 0, 2024-09-13T12:11:45.852 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_kb": 0, 2024-09-13T12:11:45.852 INFO:tasks.workunit.client.0.smithi032.stdout: "num_promote": 0, 2024-09-13T12:11:45.852 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_mode_high": 0, 2024-09-13T12:11:45.852 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_mode_low": 0, 2024-09-13T12:11:45.852 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_mode_some": 0, 2024-09-13T12:11:45.852 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_mode_full": 0, 2024-09-13T12:11:45.852 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_pinned": 0, 2024-09-13T12:11:45.852 INFO:tasks.workunit.client.0.smithi032.stdout: "num_legacy_snapsets": 0, 2024-09-13T12:11:45.852 INFO:tasks.workunit.client.0.smithi032.stdout: "num_large_omap_objects": 0, 2024-09-13T12:11:45.852 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_manifest": 0, 2024-09-13T12:11:45.853 INFO:tasks.workunit.client.0.smithi032.stdout: "num_omap_bytes": 0, 2024-09-13T12:11:45.853 INFO:tasks.workunit.client.0.smithi032.stdout: "num_omap_keys": 0, 2024-09-13T12:11:45.853 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_repaired": 0 2024-09-13T12:11:45.853 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.853 INFO:tasks.workunit.client.0.smithi032.stdout: "store_stats": { 2024-09-13T12:11:45.853 INFO:tasks.workunit.client.0.smithi032.stdout: "total": 0, 2024-09-13T12:11:45.853 INFO:tasks.workunit.client.0.smithi032.stdout: "available": 0, 2024-09-13T12:11:45.853 INFO:tasks.workunit.client.0.smithi032.stdout: "internally_reserved": 0, 2024-09-13T12:11:45.853 INFO:tasks.workunit.client.0.smithi032.stdout: "allocated": 0, 2024-09-13T12:11:45.853 INFO:tasks.workunit.client.0.smithi032.stdout: "data_stored": 0, 2024-09-13T12:11:45.853 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed": 0, 2024-09-13T12:11:45.853 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_allocated": 0, 2024-09-13T12:11:45.854 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_original": 0, 2024-09-13T12:11:45.854 INFO:tasks.workunit.client.0.smithi032.stdout: "omap_allocated": 0, 2024-09-13T12:11:45.854 INFO:tasks.workunit.client.0.smithi032.stdout: "internal_metadata": 0 2024-09-13T12:11:45.854 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.854 INFO:tasks.workunit.client.0.smithi032.stdout: "log_size": 474, 2024-09-13T12:11:45.854 INFO:tasks.workunit.client.0.smithi032.stdout: "ondisk_log_size": 474, 2024-09-13T12:11:45.854 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 64, 2024-09-13T12:11:45.854 INFO:tasks.workunit.client.0.smithi032.stdout: "acting": 64, 2024-09-13T12:11:45.854 INFO:tasks.workunit.client.0.smithi032.stdout: "num_store_stats": 0 2024-09-13T12:11:45.854 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.854 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_sum": { 2024-09-13T12:11:45.855 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 0, 2024-09-13T12:11:45.855 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 0, 2024-09-13T12:11:45.855 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pgs": 96, 2024-09-13T12:11:45.855 INFO:tasks.workunit.client.0.smithi032.stdout: "num_osds": 3, 2024-09-13T12:11:45.855 INFO:tasks.workunit.client.0.smithi032.stdout: "num_per_pool_osds": 3, 2024-09-13T12:11:45.855 INFO:tasks.workunit.client.0.smithi032.stdout: "num_per_pool_omap_osds": 3, 2024-09-13T12:11:45.855 INFO:tasks.workunit.client.0.smithi032.stdout: "kb": 314572800, 2024-09-13T12:11:45.855 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used": 84380, 2024-09-13T12:11:45.855 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_data": 2832, 2024-09-13T12:11:45.855 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_omap": 10, 2024-09-13T12:11:45.855 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_meta": 81461, 2024-09-13T12:11:45.855 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_avail": 314488420, 2024-09-13T12:11:45.855 INFO:tasks.workunit.client.0.smithi032.stdout: "statfs": { 2024-09-13T12:11:45.856 INFO:tasks.workunit.client.0.smithi032.stdout: "total": 322122547200, 2024-09-13T12:11:45.856 INFO:tasks.workunit.client.0.smithi032.stdout: "available": 322036142080, 2024-09-13T12:11:45.856 INFO:tasks.workunit.client.0.smithi032.stdout: "internally_reserved": 0, 2024-09-13T12:11:45.856 INFO:tasks.workunit.client.0.smithi032.stdout: "allocated": 2899968, 2024-09-13T12:11:45.856 INFO:tasks.workunit.client.0.smithi032.stdout: "data_stored": 688517, 2024-09-13T12:11:45.856 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed": 0, 2024-09-13T12:11:45.856 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_allocated": 0, 2024-09-13T12:11:45.856 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_original": 0, 2024-09-13T12:11:45.856 INFO:tasks.workunit.client.0.smithi032.stdout: "omap_allocated": 10396, 2024-09-13T12:11:45.856 INFO:tasks.workunit.client.0.smithi032.stdout: "internal_metadata": 83416932 2024-09-13T12:11:45.856 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.857 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_peers": [], 2024-09-13T12:11:45.857 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_trim_queue_len": 0, 2024-09-13T12:11:45.857 INFO:tasks.workunit.client.0.smithi032.stdout: "num_snap_trimming": 0, 2024-09-13T12:11:45.857 INFO:tasks.workunit.client.0.smithi032.stdout: "num_shards_repaired": 0, 2024-09-13T12:11:45.857 INFO:tasks.workunit.client.0.smithi032.stdout: "op_queue_age_hist": { 2024-09-13T12:11:45.857 INFO:tasks.workunit.client.0.smithi032.stdout: "histogram": [], 2024-09-13T12:11:45.857 INFO:tasks.workunit.client.0.smithi032.stdout: "upper_bound": 1 2024-09-13T12:11:45.857 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.857 INFO:tasks.workunit.client.0.smithi032.stdout: "perf_stat": { 2024-09-13T12:11:45.857 INFO:tasks.workunit.client.0.smithi032.stdout: "commit_latency_ms": 85, 2024-09-13T12:11:45.857 INFO:tasks.workunit.client.0.smithi032.stdout: "apply_latency_ms": 85, 2024-09-13T12:11:45.857 INFO:tasks.workunit.client.0.smithi032.stdout: "commit_latency_ns": 85000000, 2024-09-13T12:11:45.858 INFO:tasks.workunit.client.0.smithi032.stdout: "apply_latency_ns": 85000000 2024-09-13T12:11:45.858 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.858 INFO:tasks.workunit.client.0.smithi032.stdout: "alerts": [], 2024-09-13T12:11:45.858 INFO:tasks.workunit.client.0.smithi032.stdout: "network_ping_times": [] 2024-09-13T12:11:45.858 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.858 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_sum_by_class": { 2024-09-13T12:11:45.858 INFO:tasks.workunit.client.0.smithi032.stdout: "hdd": { 2024-09-13T12:11:45.858 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 0, 2024-09-13T12:11:45.858 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 0, 2024-09-13T12:11:45.858 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pgs": 96, 2024-09-13T12:11:45.858 INFO:tasks.workunit.client.0.smithi032.stdout: "num_osds": 3, 2024-09-13T12:11:45.858 INFO:tasks.workunit.client.0.smithi032.stdout: "num_per_pool_osds": 3, 2024-09-13T12:11:45.859 INFO:tasks.workunit.client.0.smithi032.stdout: "num_per_pool_omap_osds": 3, 2024-09-13T12:11:45.859 INFO:tasks.workunit.client.0.smithi032.stdout: "kb": 314572800, 2024-09-13T12:11:45.859 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used": 84380, 2024-09-13T12:11:45.859 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_data": 2832, 2024-09-13T12:11:45.859 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_omap": 10, 2024-09-13T12:11:45.859 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_meta": 81461, 2024-09-13T12:11:45.859 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_avail": 314488420, 2024-09-13T12:11:45.859 INFO:tasks.workunit.client.0.smithi032.stdout: "statfs": { 2024-09-13T12:11:45.859 INFO:tasks.workunit.client.0.smithi032.stdout: "total": 322122547200, 2024-09-13T12:11:45.859 INFO:tasks.workunit.client.0.smithi032.stdout: "available": 322036142080, 2024-09-13T12:11:45.859 INFO:tasks.workunit.client.0.smithi032.stdout: "internally_reserved": 0, 2024-09-13T12:11:45.859 INFO:tasks.workunit.client.0.smithi032.stdout: "allocated": 2899968, 2024-09-13T12:11:45.860 INFO:tasks.workunit.client.0.smithi032.stdout: "data_stored": 688517, 2024-09-13T12:11:45.860 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed": 0, 2024-09-13T12:11:45.860 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_allocated": 0, 2024-09-13T12:11:45.860 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_original": 0, 2024-09-13T12:11:45.860 INFO:tasks.workunit.client.0.smithi032.stdout: "omap_allocated": 10396, 2024-09-13T12:11:45.860 INFO:tasks.workunit.client.0.smithi032.stdout: "internal_metadata": 83416932 2024-09-13T12:11:45.860 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.860 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_peers": [], 2024-09-13T12:11:45.860 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_trim_queue_len": 0, 2024-09-13T12:11:45.860 INFO:tasks.workunit.client.0.smithi032.stdout: "num_snap_trimming": 0, 2024-09-13T12:11:45.860 INFO:tasks.workunit.client.0.smithi032.stdout: "num_shards_repaired": 0, 2024-09-13T12:11:45.860 INFO:tasks.workunit.client.0.smithi032.stdout: "op_queue_age_hist": { 2024-09-13T12:11:45.861 INFO:tasks.workunit.client.0.smithi032.stdout: "histogram": [], 2024-09-13T12:11:45.861 INFO:tasks.workunit.client.0.smithi032.stdout: "upper_bound": 1 2024-09-13T12:11:45.861 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.861 INFO:tasks.workunit.client.0.smithi032.stdout: "perf_stat": { 2024-09-13T12:11:45.861 INFO:tasks.workunit.client.0.smithi032.stdout: "commit_latency_ms": 85, 2024-09-13T12:11:45.861 INFO:tasks.workunit.client.0.smithi032.stdout: "apply_latency_ms": 85, 2024-09-13T12:11:45.861 INFO:tasks.workunit.client.0.smithi032.stdout: "commit_latency_ns": 85000000, 2024-09-13T12:11:45.861 INFO:tasks.workunit.client.0.smithi032.stdout: "apply_latency_ns": 85000000 2024-09-13T12:11:45.861 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.861 INFO:tasks.workunit.client.0.smithi032.stdout: "alerts": [], 2024-09-13T12:11:45.861 INFO:tasks.workunit.client.0.smithi032.stdout: "network_ping_times": [] 2024-09-13T12:11:45.861 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.862 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.862 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_stats": [ 2024-09-13T12:11:45.862 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.862 INFO:tasks.workunit.client.0.smithi032.stdout: "poolid": 1, 2024-09-13T12:11:45.862 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg": 32, 2024-09-13T12:11:45.862 INFO:tasks.workunit.client.0.smithi032.stdout: "stat_sum": { 2024-09-13T12:11:45.862 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes": 48514, 2024-09-13T12:11:45.862 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects": 103, 2024-09-13T12:11:45.862 INFO:tasks.workunit.client.0.smithi032.stdout: "num_object_clones": 0, 2024-09-13T12:11:45.862 INFO:tasks.workunit.client.0.smithi032.stdout: "num_object_copies": 309, 2024-09-13T12:11:45.862 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_missing_on_primary": 0, 2024-09-13T12:11:45.862 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_missing": 0, 2024-09-13T12:11:45.863 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_degraded": 103, 2024-09-13T12:11:45.863 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_misplaced": 0, 2024-09-13T12:11:45.863 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_unfound": 0, 2024-09-13T12:11:45.863 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_dirty": 103, 2024-09-13T12:11:45.863 INFO:tasks.workunit.client.0.smithi032.stdout: "num_whiteouts": 0, 2024-09-13T12:11:45.863 INFO:tasks.workunit.client.0.smithi032.stdout: "num_read": 0, 2024-09-13T12:11:45.863 INFO:tasks.workunit.client.0.smithi032.stdout: "num_read_kb": 0, 2024-09-13T12:11:45.863 INFO:tasks.workunit.client.0.smithi032.stdout: "num_write": 58, 2024-09-13T12:11:45.863 INFO:tasks.workunit.client.0.smithi032.stdout: "num_write_kb": 58, 2024-09-13T12:11:45.863 INFO:tasks.workunit.client.0.smithi032.stdout: "num_scrub_errors": 0, 2024-09-13T12:11:45.863 INFO:tasks.workunit.client.0.smithi032.stdout: "num_shallow_scrub_errors": 0, 2024-09-13T12:11:45.863 INFO:tasks.workunit.client.0.smithi032.stdout: "num_deep_scrub_errors": 0, 2024-09-13T12:11:45.864 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_recovered": 0, 2024-09-13T12:11:45.864 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes_recovered": 0, 2024-09-13T12:11:45.864 INFO:tasks.workunit.client.0.smithi032.stdout: "num_keys_recovered": 0, 2024-09-13T12:11:45.864 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_omap": 0, 2024-09-13T12:11:45.864 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_hit_set_archive": 0, 2024-09-13T12:11:45.864 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes_hit_set_archive": 0, 2024-09-13T12:11:45.864 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush": 0, 2024-09-13T12:11:45.864 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_kb": 0, 2024-09-13T12:11:45.864 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict": 0, 2024-09-13T12:11:45.864 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_kb": 0, 2024-09-13T12:11:45.864 INFO:tasks.workunit.client.0.smithi032.stdout: "num_promote": 0, 2024-09-13T12:11:45.864 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_mode_high": 0, 2024-09-13T12:11:45.865 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_mode_low": 0, 2024-09-13T12:11:45.865 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_mode_some": 0, 2024-09-13T12:11:45.865 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_mode_full": 0, 2024-09-13T12:11:45.865 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_pinned": 0, 2024-09-13T12:11:45.865 INFO:tasks.workunit.client.0.smithi032.stdout: "num_legacy_snapsets": 0, 2024-09-13T12:11:45.865 INFO:tasks.workunit.client.0.smithi032.stdout: "num_large_omap_objects": 0, 2024-09-13T12:11:45.865 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_manifest": 0, 2024-09-13T12:11:45.865 INFO:tasks.workunit.client.0.smithi032.stdout: "num_omap_bytes": 0, 2024-09-13T12:11:45.865 INFO:tasks.workunit.client.0.smithi032.stdout: "num_omap_keys": 0, 2024-09-13T12:11:45.865 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_repaired": 0 2024-09-13T12:11:45.866 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.866 INFO:tasks.workunit.client.0.smithi032.stdout: "store_stats": { 2024-09-13T12:11:45.866 INFO:tasks.workunit.client.0.smithi032.stdout: "total": 0, 2024-09-13T12:11:45.866 INFO:tasks.workunit.client.0.smithi032.stdout: "available": 0, 2024-09-13T12:11:45.866 INFO:tasks.workunit.client.0.smithi032.stdout: "internally_reserved": 0, 2024-09-13T12:11:45.866 INFO:tasks.workunit.client.0.smithi032.stdout: "allocated": 1220608, 2024-09-13T12:11:45.866 INFO:tasks.workunit.client.0.smithi032.stdout: "data_stored": 142618, 2024-09-13T12:11:45.866 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed": 0, 2024-09-13T12:11:45.866 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_allocated": 0, 2024-09-13T12:11:45.866 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_original": 0, 2024-09-13T12:11:45.866 INFO:tasks.workunit.client.0.smithi032.stdout: "omap_allocated": 0, 2024-09-13T12:11:45.866 INFO:tasks.workunit.client.0.smithi032.stdout: "internal_metadata": 0 2024-09-13T12:11:45.867 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.867 INFO:tasks.workunit.client.0.smithi032.stdout: "log_size": 474, 2024-09-13T12:11:45.867 INFO:tasks.workunit.client.0.smithi032.stdout: "ondisk_log_size": 474, 2024-09-13T12:11:45.867 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 64, 2024-09-13T12:11:45.867 INFO:tasks.workunit.client.0.smithi032.stdout: "acting": 64, 2024-09-13T12:11:45.867 INFO:tasks.workunit.client.0.smithi032.stdout: "num_store_stats": 3 2024-09-13T12:11:45.867 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.867 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:45.867 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_stats": [ 2024-09-13T12:11:45.867 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.867 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 0, 2024-09-13T12:11:45.868 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 270582939669 2024-09-13T12:11:45.868 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.868 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.868 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 1, 2024-09-13T12:11:45.868 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 42949672988 2024-09-13T12:11:45.868 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.868 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.868 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2, 2024-09-13T12:11:45.868 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 279172874260 2024-09-13T12:11:45.868 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.868 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:45.868 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg_by_state": [ 2024-09-13T12:11:45.869 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.869 INFO:tasks.workunit.client.0.smithi032.stdout: "state": "active+undersized+degraded", 2024-09-13T12:11:45.869 INFO:tasks.workunit.client.0.smithi032.stdout: "num": 32 2024-09-13T12:11:45.869 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.869 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:45.869 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg_by_osd": [ 2024-09-13T12:11:45.869 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.869 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2, 2024-09-13T12:11:45.869 INFO:tasks.workunit.client.0.smithi032.stdout: "num_primary_pg": 18, 2024-09-13T12:11:45.869 INFO:tasks.workunit.client.0.smithi032.stdout: "num_acting_pg": 32, 2024-09-13T12:11:45.870 INFO:tasks.workunit.client.0.smithi032.stdout: "num_up_not_acting_pg": 0 2024-09-13T12:11:45.870 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.870 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.870 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 0, 2024-09-13T12:11:45.870 INFO:tasks.workunit.client.0.smithi032.stdout: "num_primary_pg": 14, 2024-09-13T12:11:45.870 INFO:tasks.workunit.client.0.smithi032.stdout: "num_acting_pg": 32, 2024-09-13T12:11:45.870 INFO:tasks.workunit.client.0.smithi032.stdout: "num_up_not_acting_pg": 0 2024-09-13T12:11:45.870 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.870 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.870 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 1, 2024-09-13T12:11:45.870 INFO:tasks.workunit.client.0.smithi032.stdout: "num_primary_pg": 0, 2024-09-13T12:11:45.870 INFO:tasks.workunit.client.0.smithi032.stdout: "num_acting_pg": 0, 2024-09-13T12:11:45.871 INFO:tasks.workunit.client.0.smithi032.stdout: "num_up_not_acting_pg": 0 2024-09-13T12:11:45.871 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.871 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:45.871 INFO:tasks.workunit.client.0.smithi032.stdout: "purged_snaps": [ 2024-09-13T12:11:45.871 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:11:45.871 INFO:tasks.workunit.client.0.smithi032.stdout: "pool": 1, 2024-09-13T12:11:45.871 INFO:tasks.workunit.client.0.smithi032.stdout: "purged_snaps": {} 2024-09-13T12:11:45.871 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.871 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:11:45.871 INFO:tasks.workunit.client.0.smithi032.stdout: "servicemap": { 2024-09-13T12:11:45.871 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 7, 2024-09-13T12:11:45.872 INFO:tasks.workunit.client.0.smithi032.stdout: "modified": "2024-09-13T12:11:30.390575+0000", 2024-09-13T12:11:45.872 INFO:tasks.workunit.client.0.smithi032.stdout: "services": {} 2024-09-13T12:11:45.872 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:11:45.872 INFO:tasks.workunit.client.0.smithi032.stdout: "mgrstat_first_committed": 1, 2024-09-13T12:11:45.872 INFO:tasks.workunit.client.0.smithi032.stdout: "mgrstat_last_committed": 114, 2024-09-13T12:11:45.872 INFO:tasks.workunit.client.0.smithi032.stdout: "logm_first_committed": 1, 2024-09-13T12:11:45.872 INFO:tasks.workunit.client.0.smithi032.stdout: "logm_last_committed": 498, 2024-09-13T12:11:45.872 INFO:tasks.workunit.client.0.smithi032.stdout: "paxos": { 2024-09-13T12:11:45.872 INFO:tasks.workunit.client.0.smithi032.stdout: "first_committed": 1, 2024-09-13T12:11:45.872 INFO:tasks.workunit.client.0.smithi032.stdout: "last_committed": 677, 2024-09-13T12:11:45.873 INFO:tasks.workunit.client.0.smithi032.stdout: "last_pn": 0, 2024-09-13T12:11:45.873 INFO:tasks.workunit.client.0.smithi032.stdout: "accepted_pn": 0 2024-09-13T12:11:45.873 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:11:45.873 INFO:tasks.workunit.client.0.smithi032.stdout:} 2024-09-13T12:11:45.873 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-13T12:11:45.874 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: echo 'delay recovery' 2024-09-13T12:11:45.874 INFO:tasks.workunit.client.0.smithi032.stdout:delay recovery 2024-09-13T12:11:45.874 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph pg dump pgs 2024-09-13T12:11:46.183 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:11:46.183 INFO:tasks.workunit.client.0.smithi032.stdout:1.1f 3 0 3 0 0 1615 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422575+0000 38'45 73:109 [2,0] 2 [2,0] 2 38'45 2024-09-13T12:09:53.714380+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:46.184 INFO:tasks.workunit.client.0.smithi032.stdout:1.1e 3 0 3 0 0 1615 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.291074+0000 44'45 73:117 [0,2] 0 [0,2] 0 44'45 2024-09-13T12:10:01.695428+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:46.184 INFO:tasks.workunit.client.0.smithi032.stdout:1.1d 4 0 4 0 0 1932 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:32.422167+0000 39'46 73:121 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:49.397412+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T15:29:37.811752+0000 2 0 2024-09-13T12:11:46.184 INFO:tasks.workunit.client.0.smithi032.stdout:1.b 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.290838+0000 42'44 73:117 [0,2] 0 [0,2] 0 42'44 2024-09-13T12:09:57.774856+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:10:36.425691+0000 0 0 2024-09-13T12:11:46.184 INFO:tasks.workunit.client.0.smithi032.stdout:1.a 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422529+0000 34'44 73:92 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:50.853058+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:46.184 INFO:tasks.workunit.client.0.smithi032.stdout:1.9 5 0 5 0 0 2415 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.036386+0000 42'45 73:132 [2,0] 2 [2,0] 2 42'45 2024-09-13T12:10:03.691408+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T17:45:19.630728+0000 4 0 2024-09-13T12:11:46.184 INFO:tasks.workunit.client.0.smithi032.stdout:1.8 5 0 5 0 0 2098 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:32.422809+0000 42'46 73:115 [2,0] 2 [2,0] 2 42'46 2024-09-13T12:09:58.652148+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:46.184 INFO:tasks.workunit.client.0.smithi032.stdout:1.7 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422184+0000 34'44 73:109 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:55.719570+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:46.184 INFO:tasks.workunit.client.0.smithi032.stdout:1.6 5 0 5 0 0 2098 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291701+0000 41'46 73:130 [0,2] 0 [0,2] 0 41'46 2024-09-13T12:10:02.648875+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:46.184 INFO:tasks.workunit.client.0.smithi032.stdout:1.5 6 0 6 0 0 2898 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:32.422143+0000 42'46 73:118 [2,0] 2 [2,0] 2 38'45 2024-09-13T12:09:52.375427+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T21:28:19.785529+0000 5 0 2024-09-13T12:11:46.184 INFO:tasks.workunit.client.0.smithi032.stdout:1.4 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.290860+0000 34'44 73:100 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:46.701189+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:46.184 INFO:tasks.workunit.client.0.smithi032.stdout:1.3 5 0 5 0 0 2415 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.363328+0000 44'45 73:123 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:44.638103+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T20:31:20.891228+0000 3 0 2024-09-13T12:11:46.185 INFO:tasks.workunit.client.0.smithi032.stdout:1.2 1 0 1 0 0 483 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.291679+0000 39'44 73:118 [0,2] 0 [0,2] 0 39'44 2024-09-13T12:09:55.946588+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T18:12:36.231144+0000 1 0 2024-09-13T12:11:46.185 INFO:tasks.workunit.client.0.smithi032.stdout:1.c 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.290976+0000 41'45 73:127 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:51.776896+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:19:05.496290+0000 2 0 2024-09-13T12:11:46.185 INFO:tasks.workunit.client.0.smithi032.stdout:1.1 2 0 2 0 0 966 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422961+0000 41'45 73:114 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:48.455862+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:18:33.965292+0000 0 0 2024-09-13T12:11:46.185 INFO:tasks.workunit.client.0.smithi032.stdout:1.1c 2 0 2 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.291485+0000 34'44 73:99 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:49.691404+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:46.185 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 3 0 3 0 0 1449 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.291469+0000 42'44 73:129 [0,2] 0 [0,2] 0 32'43 2024-09-13T12:09:43.710370+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:01:49.441566+0000 2 0 2024-09-13T12:11:46.185 INFO:tasks.workunit.client.0.smithi032.stdout:1.d 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422420+0000 42'45 73:117 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:50.728150+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:24:12.357955+0000 1 0 2024-09-13T12:11:46.185 INFO:tasks.workunit.client.0.smithi032.stdout:1.e 4 0 4 0 0 1603 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291323+0000 73'46 73:121 [0,2] 0 [0,2] 0 32'43 2024-09-13T12:09:49.907160+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:38:43.613716+0000 2 0 2024-09-13T12:11:46.185 INFO:tasks.workunit.client.0.smithi032.stdout:1.f 2 0 2 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.363578+0000 38'44 73:97 [2,0] 2 [2,0] 2 38'44 2024-09-13T12:09:59.616191+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:46.185 INFO:tasks.workunit.client.0.smithi032.stdout:1.10 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.291344+0000 43'45 73:133 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:52.759350+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T12:46:25.509739+0000 2 0 2024-09-13T12:11:46.185 INFO:tasks.workunit.client.0.smithi032.stdout:1.11 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.363401+0000 34'45 73:124 [2,0] 2 [2,0] 2 34'45 2024-09-13T12:09:53.421579+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T15:04:47.312019+0000 3 0 2024-09-13T12:11:46.185 INFO:tasks.workunit.client.0.smithi032.stdout:1.12 1 0 1 0 0 483 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.290912+0000 34'44 73:123 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:52.858658+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T23:50:26.431309+0000 1 0 2024-09-13T12:11:46.185 INFO:tasks.workunit.client.0.smithi032.stdout:1.13 3 0 3 0 0 1615 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.939197+0000 44'45 73:120 [2,0] 2 [2,0] 2 44'45 2024-09-13T12:10:04.717933+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:10:37.072583+0000 0 0 2024-09-13T12:11:46.186 INFO:tasks.workunit.client.0.smithi032.stdout:1.14 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.291845+0000 44'45 73:135 [0,2] 0 [0,2] 0 32'43 2024-09-13T12:09:50.727967+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T18:54:37.906181+0000 1 0 2024-09-13T12:11:46.186 INFO:tasks.workunit.client.0.smithi032.stdout:1.15 3 0 3 0 0 1449 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422194+0000 38'44 73:112 [2,0] 2 [2,0] 2 38'44 2024-09-13T12:09:54.402988+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:26:56.754038+0000 3 0 2024-09-13T12:11:46.186 INFO:tasks.workunit.client.0.smithi032.stdout:1.16 4 0 4 0 0 1932 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291960+0000 38'46 73:138 [0,2] 0 [0,2] 0 38'46 2024-09-13T12:10:00.648885+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:11:39.586367+0000 4 0 2024-09-13T12:11:46.186 INFO:tasks.workunit.client.0.smithi032.stdout:1.17 1 0 1 0 0 649 0 0 13 30 13 active+undersized+degraded 2024-09-13T12:10:32.423246+0000 32'43 73:96 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:56.659002+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:46.186 INFO:tasks.workunit.client.0.smithi032.stdout:1.18 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422567+0000 44'45 73:127 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:47.765518+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T12:54:37.159964+0000 1 0 2024-09-13T12:11:46.186 INFO:tasks.workunit.client.0.smithi032.stdout:1.19 2 0 2 0 0 966 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422893+0000 34'44 73:139 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:51.375338+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:16:57.418519+0000 2 0 2024-09-13T12:11:46.186 INFO:tasks.workunit.client.0.smithi032.stdout:1.1a 4 0 4 0 0 1932 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.423351+0000 41'45 73:119 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:51.953415+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T21:57:52.925654+0000 3 0 2024-09-13T12:11:46.186 INFO:tasks.workunit.client.0.smithi032.stdout:1.1b 5 0 5 0 0 2415 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291329+0000 44'46 73:140 [0,2] 0 [0,2] 0 36'45 2024-09-13T12:09:54.677751+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T15:59:12.687663+0000 4 0 2024-09-13T12:11:46.186 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:11:46.186 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:11:46.186 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:11:46.195 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-09-13T12:11:46.196 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.0 2024-09-13T12:11:46.196 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-13T12:11:46.196 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-13T12:11:46.196 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:11:46.196 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:11:46.196 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:11:46.196 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-osd.0.asok 2024-09-13T12:11:46.197 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-09-13T12:11:46.197 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.33242/ceph-osd.0.asok set_recovery_delay 100000 2024-09-13T12:11:46.281 INFO:tasks.workunit.client.0.smithi032.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-13T12:11:46.288 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-09-13T12:11:46.288 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.2 2024-09-13T12:11:46.288 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-13T12:11:46.288 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-13T12:11:46.289 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:11:46.290 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:11:46.290 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:11:46.290 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-osd.2.asok 2024-09-13T12:11:46.290 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-09-13T12:11:46.290 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.33242/ceph-osd.2.asok set_recovery_delay 100000 2024-09-13T12:11:46.374 INFO:tasks.workunit.client.0.smithi032.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-13T12:11:46.381 INFO:tasks.workunit.client.0.smithi032.stdout:revive divergent 1 2024-09-13T12:11:46.382 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:769: TEST_divergent_3: echo 'revive divergent 1' 2024-09-13T12:11:46.382 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2024-09-13T12:11:46.798 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:11:46.798 INFO:tasks.workunit.client.0.smithi032.stdout:1.1f 3 0 3 0 0 1615 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422575+0000 38'45 73:109 [2,0] 2 [2,0] 2 38'45 2024-09-13T12:09:53.714380+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:46.798 INFO:tasks.workunit.client.0.smithi032.stdout:1.1e 3 0 3 0 0 1615 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.291074+0000 44'45 73:117 [0,2] 0 [0,2] 0 44'45 2024-09-13T12:10:01.695428+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:46.798 INFO:tasks.workunit.client.0.smithi032.stdout:1.1d 4 0 4 0 0 1932 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:32.422167+0000 39'46 73:121 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:49.397412+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T15:29:37.811752+0000 2 0 2024-09-13T12:11:46.798 INFO:tasks.workunit.client.0.smithi032.stdout:1.b 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.290838+0000 42'44 73:117 [0,2] 0 [0,2] 0 42'44 2024-09-13T12:09:57.774856+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:10:36.425691+0000 0 0 2024-09-13T12:11:46.798 INFO:tasks.workunit.client.0.smithi032.stdout:1.a 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422529+0000 34'44 73:92 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:50.853058+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:46.798 INFO:tasks.workunit.client.0.smithi032.stdout:1.9 5 0 5 0 0 2415 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.036386+0000 42'45 73:132 [2,0] 2 [2,0] 2 42'45 2024-09-13T12:10:03.691408+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T17:45:19.630728+0000 4 0 2024-09-13T12:11:46.798 INFO:tasks.workunit.client.0.smithi032.stdout:1.8 5 0 5 0 0 2098 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:32.422809+0000 42'46 73:115 [2,0] 2 [2,0] 2 42'46 2024-09-13T12:09:58.652148+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:46.799 INFO:tasks.workunit.client.0.smithi032.stdout:1.7 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422184+0000 34'44 73:109 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:55.719570+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:46.799 INFO:tasks.workunit.client.0.smithi032.stdout:1.6 5 0 5 0 0 2098 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291701+0000 41'46 73:130 [0,2] 0 [0,2] 0 41'46 2024-09-13T12:10:02.648875+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:46.799 INFO:tasks.workunit.client.0.smithi032.stdout:1.5 6 0 6 0 0 2898 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:32.422143+0000 42'46 73:118 [2,0] 2 [2,0] 2 38'45 2024-09-13T12:09:52.375427+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T21:28:19.785529+0000 5 0 2024-09-13T12:11:46.799 INFO:tasks.workunit.client.0.smithi032.stdout:1.4 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.290860+0000 34'44 73:100 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:46.701189+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:46.799 INFO:tasks.workunit.client.0.smithi032.stdout:1.3 5 0 5 0 0 2415 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.363328+0000 44'45 73:123 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:44.638103+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T20:31:20.891228+0000 3 0 2024-09-13T12:11:46.799 INFO:tasks.workunit.client.0.smithi032.stdout:1.2 1 0 1 0 0 483 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.291679+0000 39'44 73:118 [0,2] 0 [0,2] 0 39'44 2024-09-13T12:09:55.946588+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T18:12:36.231144+0000 1 0 2024-09-13T12:11:46.799 INFO:tasks.workunit.client.0.smithi032.stdout:1.c 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.290976+0000 41'45 73:127 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:51.776896+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:19:05.496290+0000 2 0 2024-09-13T12:11:46.799 INFO:tasks.workunit.client.0.smithi032.stdout:1.1 2 0 2 0 0 966 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422961+0000 41'45 73:114 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:48.455862+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:18:33.965292+0000 0 0 2024-09-13T12:11:46.799 INFO:tasks.workunit.client.0.smithi032.stdout:1.1c 2 0 2 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.291485+0000 34'44 73:99 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:49.691404+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:46.799 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 3 0 3 0 0 1449 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.291469+0000 42'44 73:129 [0,2] 0 [0,2] 0 32'43 2024-09-13T12:09:43.710370+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:01:49.441566+0000 2 0 2024-09-13T12:11:46.799 INFO:tasks.workunit.client.0.smithi032.stdout:1.d 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422420+0000 42'45 73:117 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:50.728150+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:24:12.357955+0000 1 0 2024-09-13T12:11:46.799 INFO:tasks.workunit.client.0.smithi032.stdout:1.e 4 0 4 0 0 1603 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291323+0000 73'46 73:121 [0,2] 0 [0,2] 0 32'43 2024-09-13T12:09:49.907160+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:38:43.613716+0000 2 0 2024-09-13T12:11:46.799 INFO:tasks.workunit.client.0.smithi032.stdout:1.f 2 0 2 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.363578+0000 38'44 73:97 [2,0] 2 [2,0] 2 38'44 2024-09-13T12:09:59.616191+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:46.799 INFO:tasks.workunit.client.0.smithi032.stdout:1.10 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.291344+0000 43'45 73:133 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:52.759350+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T12:46:25.509739+0000 2 0 2024-09-13T12:11:46.800 INFO:tasks.workunit.client.0.smithi032.stdout:1.11 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.363401+0000 34'45 73:124 [2,0] 2 [2,0] 2 34'45 2024-09-13T12:09:53.421579+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T15:04:47.312019+0000 3 0 2024-09-13T12:11:46.800 INFO:tasks.workunit.client.0.smithi032.stdout:1.12 1 0 1 0 0 483 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.290912+0000 34'44 73:123 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:52.858658+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T23:50:26.431309+0000 1 0 2024-09-13T12:11:46.800 INFO:tasks.workunit.client.0.smithi032.stdout:1.13 3 0 3 0 0 1615 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.939197+0000 44'45 73:120 [2,0] 2 [2,0] 2 44'45 2024-09-13T12:10:04.717933+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:10:37.072583+0000 0 0 2024-09-13T12:11:46.800 INFO:tasks.workunit.client.0.smithi032.stdout:1.14 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.291845+0000 44'45 73:135 [0,2] 0 [0,2] 0 32'43 2024-09-13T12:09:50.727967+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T18:54:37.906181+0000 1 0 2024-09-13T12:11:46.800 INFO:tasks.workunit.client.0.smithi032.stdout:1.15 3 0 3 0 0 1449 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422194+0000 38'44 73:112 [2,0] 2 [2,0] 2 38'44 2024-09-13T12:09:54.402988+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:26:56.754038+0000 3 0 2024-09-13T12:11:46.800 INFO:tasks.workunit.client.0.smithi032.stdout:1.16 4 0 4 0 0 1932 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291960+0000 38'46 73:138 [0,2] 0 [0,2] 0 38'46 2024-09-13T12:10:00.648885+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:11:39.586367+0000 4 0 2024-09-13T12:11:46.800 INFO:tasks.workunit.client.0.smithi032.stdout:1.17 1 0 1 0 0 649 0 0 13 30 13 active+undersized+degraded 2024-09-13T12:10:32.423246+0000 32'43 73:96 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:56.659002+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:46.800 INFO:tasks.workunit.client.0.smithi032.stdout:1.18 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422567+0000 44'45 73:127 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:47.765518+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T12:54:37.159964+0000 1 0 2024-09-13T12:11:46.800 INFO:tasks.workunit.client.0.smithi032.stdout:1.19 2 0 2 0 0 966 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422893+0000 34'44 73:139 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:51.375338+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:16:57.418519+0000 2 0 2024-09-13T12:11:46.800 INFO:tasks.workunit.client.0.smithi032.stdout:1.1a 4 0 4 0 0 1932 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.423351+0000 41'45 73:119 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:51.953415+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T21:57:52.925654+0000 3 0 2024-09-13T12:11:46.800 INFO:tasks.workunit.client.0.smithi032.stdout:1.1b 5 0 5 0 0 2415 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291329+0000 44'46 73:140 [0,2] 0 [0,2] 0 36'45 2024-09-13T12:09:54.677751+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T15:59:12.687663+0000 4 0 2024-09-13T12:11:46.800 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:11:46.800 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:11:46.801 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:11:46.809 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: ceph osd set noup 2024-09-13T12:11:47.484 INFO:tasks.workunit.client.0.smithi032.stderr:noup is set 2024-09-13T12:11:47.498 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: activate_osd td/divergent-priors 1 2024-09-13T12:11:47.498 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-13T12:11:47.499 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:11:47.499 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T12:11:47.499 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:11:47.499 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-13T12:11:47.499 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T12:11:47.499 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:11:47.499 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:11:47.499 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:11:47.499 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-13T12:11:47.500 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-13T12:11:47.500 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:11:47.500 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:11:47.500 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T12:11:47.500 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:11:47.500 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:11:47.500 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:11:47.500 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:11:47.500 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:11:47.500 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:11:47.500 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:11:47.501 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:11:47.501 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:11:47.501 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T12:11:47.501 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T12:11:47.501 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:11:47.501 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:11:47.501 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:11:47.501 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:11:47.501 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:11:47.501 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:11:47.501 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-13T12:11:47.501 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T12:11:47.502 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.1 2024-09-13T12:11:47.503 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:11:47.503 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-13T12:11:47.503 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T12:11:47.503 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:11:47.503 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:11:47.503 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:11:47.534 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:11:47.533+0000 7f749102a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:11:47.549 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:11:47.548+0000 7f749102a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:11:47.550 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:11:47.549+0000 7f749102a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:11:47.972 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: return 0 2024-09-13T12:11:47.972 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: sleep 5 2024-09-13T12:11:48.898 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:11:48.897+0000 7f749102a640 -1 Falling back to public interface 2024-09-13T12:11:52.398 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:11:52.397+0000 7f749102a640 -1 osd.1 56 log_to_monitors true 2024-09-13T12:11:52.973 INFO:tasks.workunit.client.0.smithi032.stdout:delay recovery divergent 2024-09-13T12:11:52.974 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:775: TEST_divergent_3: echo 'delay recovery divergent' 2024-09-13T12:11:52.974 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:776: TEST_divergent_3: ceph pg dump pgs 2024-09-13T12:11:53.374 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:11:53.374 INFO:tasks.workunit.client.0.smithi032.stdout:1.1f 3 0 3 0 0 1615 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422575+0000 38'45 75:112 [2,0] 2 [2,0] 2 38'45 2024-09-13T12:09:53.714380+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:50.624810+0000 0 0 2024-09-13T12:11:53.374 INFO:tasks.workunit.client.0.smithi032.stdout:1.1e 3 0 3 0 0 1615 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.291074+0000 44'45 75:121 [0,2] 0 [0,2] 0 44'45 2024-09-13T12:10:01.695428+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:50.801987+0000 0 0 2024-09-13T12:11:53.374 INFO:tasks.workunit.client.0.smithi032.stdout:1.1d 4 0 4 0 0 1932 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:32.422167+0000 39'46 75:124 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:49.397412+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T15:29:37.811752+0000 2 0 2024-09-13T12:11:53.374 INFO:tasks.workunit.client.0.smithi032.stdout:1.b 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.290838+0000 42'44 75:121 [0,2] 0 [0,2] 0 42'44 2024-09-13T12:09:57.774856+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:51.792644+0000 0 0 2024-09-13T12:11:53.375 INFO:tasks.workunit.client.0.smithi032.stdout:1.a 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422529+0000 34'44 75:95 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:50.853058+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:48.694120+0000 0 0 2024-09-13T12:11:53.375 INFO:tasks.workunit.client.0.smithi032.stdout:1.9 5 0 5 0 0 2415 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.036386+0000 42'45 75:135 [2,0] 2 [2,0] 2 42'45 2024-09-13T12:10:03.691408+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T17:45:19.630728+0000 4 0 2024-09-13T12:11:53.375 INFO:tasks.workunit.client.0.smithi032.stdout:1.8 5 0 5 0 0 2098 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:32.422809+0000 42'46 75:118 [2,0] 2 [2,0] 2 42'46 2024-09-13T12:09:58.652148+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:51.653169+0000 0 0 2024-09-13T12:11:53.375 INFO:tasks.workunit.client.0.smithi032.stdout:1.7 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422184+0000 34'44 75:112 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:55.719570+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:47.659938+0000 0 0 2024-09-13T12:11:53.375 INFO:tasks.workunit.client.0.smithi032.stdout:1.6 5 0 5 0 0 2098 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291701+0000 41'46 75:134 [0,2] 0 [0,2] 0 41'46 2024-09-13T12:10:02.648875+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:49.769629+0000 0 0 2024-09-13T12:11:53.375 INFO:tasks.workunit.client.0.smithi032.stdout:1.5 6 0 6 0 0 2898 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:32.422143+0000 42'46 75:121 [2,0] 2 [2,0] 2 38'45 2024-09-13T12:09:52.375427+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T21:28:19.785529+0000 5 0 2024-09-13T12:11:53.375 INFO:tasks.workunit.client.0.smithi032.stdout:1.4 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.290860+0000 34'44 75:104 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:46.701189+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:53.375 INFO:tasks.workunit.client.0.smithi032.stdout:1.3 5 0 5 0 0 2415 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.363328+0000 44'45 75:126 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:44.638103+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T20:31:20.891228+0000 3 0 2024-09-13T12:11:53.375 INFO:tasks.workunit.client.0.smithi032.stdout:1.2 1 0 1 0 0 483 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.291679+0000 39'44 75:122 [0,2] 0 [0,2] 0 39'44 2024-09-13T12:09:55.946588+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T18:12:36.231144+0000 1 0 2024-09-13T12:11:53.375 INFO:tasks.workunit.client.0.smithi032.stdout:1.c 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.290976+0000 41'45 75:131 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:51.776896+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:19:05.496290+0000 2 0 2024-09-13T12:11:53.375 INFO:tasks.workunit.client.0.smithi032.stdout:1.1 2 0 2 0 0 966 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422961+0000 41'45 75:117 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:48.455862+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:18:33.965292+0000 0 0 2024-09-13T12:11:53.375 INFO:tasks.workunit.client.0.smithi032.stdout:1.1c 2 0 2 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.291485+0000 34'44 75:103 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:49.691404+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:48.740642+0000 0 0 2024-09-13T12:11:53.375 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 3 0 3 0 0 1449 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.291469+0000 42'44 75:133 [0,2] 0 [0,2] 0 32'43 2024-09-13T12:09:43.710370+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:01:49.441566+0000 2 0 2024-09-13T12:11:53.376 INFO:tasks.workunit.client.0.smithi032.stdout:1.d 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422420+0000 42'45 75:120 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:50.728150+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:24:12.357955+0000 1 0 2024-09-13T12:11:53.376 INFO:tasks.workunit.client.0.smithi032.stdout:1.e 4 0 4 0 0 1603 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291323+0000 73'46 75:125 [0,2] 0 [0,2] 0 32'43 2024-09-13T12:09:49.907160+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:38:43.613716+0000 2 0 2024-09-13T12:11:53.376 INFO:tasks.workunit.client.0.smithi032.stdout:1.f 2 0 2 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.363578+0000 38'44 75:100 [2,0] 2 [2,0] 2 38'44 2024-09-13T12:09:59.616191+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:53.376 INFO:tasks.workunit.client.0.smithi032.stdout:1.10 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.291344+0000 43'45 75:137 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:52.759350+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T12:46:25.509739+0000 2 0 2024-09-13T12:11:53.376 INFO:tasks.workunit.client.0.smithi032.stdout:1.11 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.363401+0000 34'45 75:127 [2,0] 2 [2,0] 2 34'45 2024-09-13T12:09:53.421579+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T15:04:47.312019+0000 3 0 2024-09-13T12:11:53.376 INFO:tasks.workunit.client.0.smithi032.stdout:1.12 1 0 1 0 0 483 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.290912+0000 34'44 75:127 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:52.858658+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T23:50:26.431309+0000 1 0 2024-09-13T12:11:53.376 INFO:tasks.workunit.client.0.smithi032.stdout:1.13 3 0 3 0 0 1615 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.939197+0000 44'45 75:123 [2,0] 2 [2,0] 2 44'45 2024-09-13T12:10:04.717933+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:49.654578+0000 0 0 2024-09-13T12:11:53.376 INFO:tasks.workunit.client.0.smithi032.stdout:1.14 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.291845+0000 44'45 75:139 [0,2] 0 [0,2] 0 32'43 2024-09-13T12:09:50.727967+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T18:54:37.906181+0000 1 0 2024-09-13T12:11:53.376 INFO:tasks.workunit.client.0.smithi032.stdout:1.15 3 0 3 0 0 1449 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422194+0000 38'44 75:115 [2,0] 2 [2,0] 2 38'44 2024-09-13T12:09:54.402988+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:26:56.754038+0000 3 0 2024-09-13T12:11:53.376 INFO:tasks.workunit.client.0.smithi032.stdout:1.16 4 0 4 0 0 1932 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291960+0000 38'46 75:142 [0,2] 0 [0,2] 0 38'46 2024-09-13T12:10:00.648885+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:11:39.586367+0000 4 0 2024-09-13T12:11:53.376 INFO:tasks.workunit.client.0.smithi032.stdout:1.17 1 0 1 0 0 649 0 0 13 30 13 active+undersized+degraded 2024-09-13T12:10:32.423246+0000 32'43 75:99 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:56.659002+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:53.376 INFO:tasks.workunit.client.0.smithi032.stdout:1.18 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422567+0000 44'45 75:130 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:47.765518+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T12:54:37.159964+0000 1 0 2024-09-13T12:11:53.377 INFO:tasks.workunit.client.0.smithi032.stdout:1.19 2 0 2 0 0 966 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422893+0000 34'44 75:142 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:51.375338+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:16:57.418519+0000 2 0 2024-09-13T12:11:53.377 INFO:tasks.workunit.client.0.smithi032.stdout:1.1a 4 0 4 0 0 1932 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.423351+0000 41'45 75:122 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:51.953415+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T21:57:52.925654+0000 3 0 2024-09-13T12:11:53.377 INFO:tasks.workunit.client.0.smithi032.stdout:1.1b 5 0 5 0 0 2415 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291329+0000 44'46 75:144 [0,2] 0 [0,2] 0 36'45 2024-09-13T12:09:54.677751+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T15:59:12.687663+0000 4 0 2024-09-13T12:11:53.377 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:11:53.377 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:11:53.377 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:11:53.386 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: get_asok_path osd.1 2024-09-13T12:11:53.386 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-13T12:11:53.386 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-13T12:11:53.387 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:11:53.387 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:11:53.387 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:11:53.387 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-osd.1.asok 2024-09-13T12:11:53.387 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: CEPH_ARGS= 2024-09-13T12:11:53.387 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.33242/ceph-osd.1.asok set_recovery_delay 100000 2024-09-13T12:11:53.471 INFO:tasks.workunit.client.0.smithi032.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-13T12:11:53.479 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: ceph osd unset noup 2024-09-13T12:11:54.136 INFO:tasks.workunit.client.0.smithi032.stderr:noup is unset 2024-09-13T12:11:54.147 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:781: TEST_divergent_3: wait_for_osd up 0 2024-09-13T12:11:54.148 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:11:54.148 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T12:11:54.148 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:11:54.148 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:11:54.148 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:11:54.148 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:11:54.149 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:11:54.149 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:11:54.149 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:11:54.743 INFO:tasks.workunit.client.0.smithi032.stdout:osd.0 up in weight 1 up_from 63 up_thru 78 down_at 57 last_clean_interval [5,50) [v2:127.0.0.1:6802/2302613541,v1:127.0.0.1:6803/2302613541] [v2:127.0.0.1:6804/2302613541,v1:127.0.0.1:6805/2302613541] exists,up 10a21bfc-e16b-4a73-8068-b8ae59576a72 2024-09-13T12:11:54.744 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:11:54.744 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:11:54.745 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:11:54.745 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: wait_for_osd up 1 2024-09-13T12:11:54.745 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:11:54.745 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T12:11:54.745 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:11:54.745 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:11:54.745 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:11:54.745 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:11:54.745 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:11:54.746 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:11:54.746 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:11:55.381 INFO:tasks.workunit.client.0.smithi032.stdout:osd.1 up in weight 1 up_from 78 up_thru 78 down_at 58 last_clean_interval [10,56) [v2:127.0.0.1:6818/108162265,v1:127.0.0.1:6819/108162265] [v2:127.0.0.1:6820/108162265,v1:127.0.0.1:6821/108162265] exists,up f377d2cc-d182-4459-bd79-b23c9379a371 2024-09-13T12:11:55.382 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:11:55.382 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:11:55.382 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:11:55.382 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: wait_for_osd up 2 2024-09-13T12:11:55.382 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:11:55.382 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T12:11:55.382 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:11:55.382 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:11:55.382 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:11:55.382 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:11:55.383 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:11:55.383 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:11:55.383 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:11:55.833 INFO:tasks.workunit.client.0.smithi032.stdout:osd.2 up in weight 1 up_from 65 up_thru 78 down_at 59 last_clean_interval [15,51) [v2:127.0.0.1:6810/73367264,v1:127.0.0.1:6811/73367264] [v2:127.0.0.1:6812/73367264,v1:127.0.0.1:6813/73367264] exists,up 6ac797ab-1924-4ff8-9169-021a47e84482 2024-09-13T12:11:55.834 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:11:55.834 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:11:55.834 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:11:55.834 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:785: TEST_divergent_3: ceph pg dump pgs 2024-09-13T12:11:56.231 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:11:56.231 INFO:tasks.workunit.client.0.smithi032.stdout:1.1f 3 0 3 0 0 1615 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422575+0000 38'45 76:114 [2,0] 2 [2,0] 2 38'45 2024-09-13T12:09:53.714380+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:56.617901+0000 0 0 2024-09-13T12:11:56.231 INFO:tasks.workunit.client.0.smithi032.stdout:1.1e 3 0 3 0 0 1615 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.291074+0000 44'45 75:121 [0,2] 0 [0,2] 0 44'45 2024-09-13T12:10:01.695428+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:50.801987+0000 0 0 2024-09-13T12:11:56.231 INFO:tasks.workunit.client.0.smithi032.stdout:1.1d 4 0 4 0 0 1932 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:32.422167+0000 39'46 76:126 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:49.397412+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T15:29:37.811752+0000 2 0 2024-09-13T12:11:56.231 INFO:tasks.workunit.client.0.smithi032.stdout:1.b 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.290838+0000 42'44 75:121 [0,2] 0 [0,2] 0 42'44 2024-09-13T12:09:57.774856+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:51.792644+0000 0 0 2024-09-13T12:11:56.231 INFO:tasks.workunit.client.0.smithi032.stdout:1.a 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422529+0000 34'44 76:97 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:50.853058+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:54.598530+0000 0 0 2024-09-13T12:11:56.231 INFO:tasks.workunit.client.0.smithi032.stdout:1.9 5 0 5 0 0 2415 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.036386+0000 42'45 76:137 [2,0] 2 [2,0] 2 42'45 2024-09-13T12:10:03.691408+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T17:45:19.630728+0000 4 0 2024-09-13T12:11:56.232 INFO:tasks.workunit.client.0.smithi032.stdout:1.8 5 0 5 0 0 2098 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:32.422809+0000 42'46 76:120 [2,0] 2 [2,0] 2 42'46 2024-09-13T12:09:58.652148+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:57.589433+0000 0 0 2024-09-13T12:11:56.232 INFO:tasks.workunit.client.0.smithi032.stdout:1.7 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422184+0000 34'44 76:114 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:55.719570+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:53.645748+0000 0 0 2024-09-13T12:11:56.232 INFO:tasks.workunit.client.0.smithi032.stdout:1.6 5 0 5 0 0 2098 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291701+0000 41'46 75:134 [0,2] 0 [0,2] 0 41'46 2024-09-13T12:10:02.648875+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:49.769629+0000 0 0 2024-09-13T12:11:56.232 INFO:tasks.workunit.client.0.smithi032.stdout:1.5 6 0 6 0 0 2898 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:32.422143+0000 42'46 76:123 [2,0] 2 [2,0] 2 38'45 2024-09-13T12:09:52.375427+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T21:28:19.785529+0000 5 0 2024-09-13T12:11:56.232 INFO:tasks.workunit.client.0.smithi032.stdout:1.4 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.290860+0000 34'44 75:104 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:46.701189+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:56.232 INFO:tasks.workunit.client.0.smithi032.stdout:1.3 5 0 5 0 0 2415 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.363328+0000 44'45 76:128 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:44.638103+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T20:31:20.891228+0000 3 0 2024-09-13T12:11:56.232 INFO:tasks.workunit.client.0.smithi032.stdout:1.2 1 0 1 0 0 483 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.291679+0000 39'44 75:122 [0,2] 0 [0,2] 0 39'44 2024-09-13T12:09:55.946588+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T18:12:36.231144+0000 1 0 2024-09-13T12:11:56.232 INFO:tasks.workunit.client.0.smithi032.stdout:1.c 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.290976+0000 41'45 75:131 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:51.776896+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:19:05.496290+0000 2 0 2024-09-13T12:11:56.232 INFO:tasks.workunit.client.0.smithi032.stdout:1.1 2 0 2 0 0 966 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422961+0000 41'45 76:119 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:48.455862+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:18:33.965292+0000 0 0 2024-09-13T12:11:56.232 INFO:tasks.workunit.client.0.smithi032.stdout:1.1c 2 0 2 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.291485+0000 34'44 75:103 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:49.691404+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:48.740642+0000 0 0 2024-09-13T12:11:56.232 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 3 0 3 0 0 1449 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.291469+0000 42'44 75:133 [0,2] 0 [0,2] 0 32'43 2024-09-13T12:09:43.710370+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:01:49.441566+0000 2 0 2024-09-13T12:11:56.232 INFO:tasks.workunit.client.0.smithi032.stdout:1.d 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422420+0000 42'45 76:122 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:50.728150+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:24:12.357955+0000 1 0 2024-09-13T12:11:56.233 INFO:tasks.workunit.client.0.smithi032.stdout:1.e 4 0 4 0 0 1603 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291323+0000 73'46 75:125 [0,2] 0 [0,2] 0 32'43 2024-09-13T12:09:49.907160+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:38:43.613716+0000 2 0 2024-09-13T12:11:56.233 INFO:tasks.workunit.client.0.smithi032.stdout:1.f 2 0 2 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.363578+0000 38'44 76:102 [2,0] 2 [2,0] 2 38'44 2024-09-13T12:09:59.616191+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:56.233 INFO:tasks.workunit.client.0.smithi032.stdout:1.10 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.291344+0000 43'45 75:137 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:52.759350+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T12:46:25.509739+0000 2 0 2024-09-13T12:11:56.233 INFO:tasks.workunit.client.0.smithi032.stdout:1.11 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.363401+0000 34'45 76:129 [2,0] 2 [2,0] 2 34'45 2024-09-13T12:09:53.421579+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T15:04:47.312019+0000 3 0 2024-09-13T12:11:56.233 INFO:tasks.workunit.client.0.smithi032.stdout:1.12 1 0 1 0 0 483 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.290912+0000 34'44 75:127 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:52.858658+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T23:50:26.431309+0000 1 0 2024-09-13T12:11:56.233 INFO:tasks.workunit.client.0.smithi032.stdout:1.13 3 0 3 0 0 1615 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.939197+0000 44'45 76:125 [2,0] 2 [2,0] 2 44'45 2024-09-13T12:10:04.717933+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:55.610436+0000 0 0 2024-09-13T12:11:56.233 INFO:tasks.workunit.client.0.smithi032.stdout:1.14 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.291845+0000 44'45 75:139 [0,2] 0 [0,2] 0 32'43 2024-09-13T12:09:50.727967+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T18:54:37.906181+0000 1 0 2024-09-13T12:11:56.233 INFO:tasks.workunit.client.0.smithi032.stdout:1.15 3 0 3 0 0 1449 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422194+0000 38'44 76:117 [2,0] 2 [2,0] 2 38'44 2024-09-13T12:09:54.402988+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:26:56.754038+0000 3 0 2024-09-13T12:11:56.233 INFO:tasks.workunit.client.0.smithi032.stdout:1.16 4 0 4 0 0 1932 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291960+0000 38'46 75:142 [0,2] 0 [0,2] 0 38'46 2024-09-13T12:10:00.648885+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:11:39.586367+0000 4 0 2024-09-13T12:11:56.233 INFO:tasks.workunit.client.0.smithi032.stdout:1.17 1 0 1 0 0 649 0 0 13 30 13 active+undersized+degraded 2024-09-13T12:10:32.423246+0000 32'43 76:101 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:56.659002+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:56.233 INFO:tasks.workunit.client.0.smithi032.stdout:1.18 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422567+0000 44'45 76:132 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:47.765518+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T12:54:37.159964+0000 1 0 2024-09-13T12:11:56.233 INFO:tasks.workunit.client.0.smithi032.stdout:1.19 2 0 2 0 0 966 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422893+0000 34'44 76:144 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:51.375338+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:16:57.418519+0000 2 0 2024-09-13T12:11:56.234 INFO:tasks.workunit.client.0.smithi032.stdout:1.1a 4 0 4 0 0 1932 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.423351+0000 41'45 76:124 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:51.953415+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T21:57:52.925654+0000 3 0 2024-09-13T12:11:56.234 INFO:tasks.workunit.client.0.smithi032.stdout:1.1b 5 0 5 0 0 2415 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291329+0000 44'46 75:144 [0,2] 0 [0,2] 0 36'45 2024-09-13T12:09:54.677751+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T15:59:12.687663+0000 4 0 2024-09-13T12:11:56.234 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:11:56.234 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:11:56.234 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:11:56.242 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'wait for peering' 2024-09-13T12:11:56.242 INFO:tasks.workunit.client.0.smithi032.stdout:wait for peering 2024-09-13T12:11:56.242 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2024-09-13T12:11:56.697 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:11:56.697 INFO:tasks.workunit.client.0.smithi032.stdout:1.1f 3 0 3 0 0 1615 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422575+0000 38'45 76:114 [2,0] 2 [2,0] 2 38'45 2024-09-13T12:09:53.714380+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:56.617901+0000 0 0 2024-09-13T12:11:56.697 INFO:tasks.workunit.client.0.smithi032.stdout:1.1e 3 0 3 0 0 1615 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.291074+0000 44'45 75:121 [0,2] 0 [0,2] 0 44'45 2024-09-13T12:10:01.695428+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:50.801987+0000 0 0 2024-09-13T12:11:56.697 INFO:tasks.workunit.client.0.smithi032.stdout:1.1d 4 0 4 0 0 1932 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:32.422167+0000 39'46 76:126 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:49.397412+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T15:29:37.811752+0000 2 0 2024-09-13T12:11:56.697 INFO:tasks.workunit.client.0.smithi032.stdout:1.b 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.290838+0000 42'44 75:121 [0,2] 0 [0,2] 0 42'44 2024-09-13T12:09:57.774856+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:51.792644+0000 0 0 2024-09-13T12:11:56.697 INFO:tasks.workunit.client.0.smithi032.stdout:1.a 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422529+0000 34'44 76:97 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:50.853058+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:54.598530+0000 0 0 2024-09-13T12:11:56.697 INFO:tasks.workunit.client.0.smithi032.stdout:1.9 5 0 5 0 0 2415 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.036386+0000 42'45 76:137 [2,0] 2 [2,0] 2 42'45 2024-09-13T12:10:03.691408+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T17:45:19.630728+0000 4 0 2024-09-13T12:11:56.697 INFO:tasks.workunit.client.0.smithi032.stdout:1.8 5 0 5 0 0 2098 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:32.422809+0000 42'46 76:120 [2,0] 2 [2,0] 2 42'46 2024-09-13T12:09:58.652148+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:57.589433+0000 0 0 2024-09-13T12:11:56.697 INFO:tasks.workunit.client.0.smithi032.stdout:1.7 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422184+0000 34'44 76:114 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:55.719570+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:53.645748+0000 0 0 2024-09-13T12:11:56.698 INFO:tasks.workunit.client.0.smithi032.stdout:1.6 5 0 5 0 0 2098 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291701+0000 41'46 75:134 [0,2] 0 [0,2] 0 41'46 2024-09-13T12:10:02.648875+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:49.769629+0000 0 0 2024-09-13T12:11:56.698 INFO:tasks.workunit.client.0.smithi032.stdout:1.5 6 0 6 0 0 2898 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:32.422143+0000 42'46 76:123 [2,0] 2 [2,0] 2 38'45 2024-09-13T12:09:52.375427+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T21:28:19.785529+0000 5 0 2024-09-13T12:11:56.698 INFO:tasks.workunit.client.0.smithi032.stdout:1.4 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.290860+0000 34'44 75:104 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:46.701189+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:56.698 INFO:tasks.workunit.client.0.smithi032.stdout:1.3 5 0 5 0 0 2415 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.363328+0000 44'45 76:128 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:44.638103+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T20:31:20.891228+0000 3 0 2024-09-13T12:11:56.698 INFO:tasks.workunit.client.0.smithi032.stdout:1.2 1 0 1 0 0 483 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.291679+0000 39'44 75:122 [0,2] 0 [0,2] 0 39'44 2024-09-13T12:09:55.946588+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T18:12:36.231144+0000 1 0 2024-09-13T12:11:56.698 INFO:tasks.workunit.client.0.smithi032.stdout:1.c 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.290976+0000 41'45 75:131 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:51.776896+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:19:05.496290+0000 2 0 2024-09-13T12:11:56.698 INFO:tasks.workunit.client.0.smithi032.stdout:1.1 2 0 2 0 0 966 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422961+0000 41'45 76:119 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:48.455862+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:18:33.965292+0000 0 0 2024-09-13T12:11:56.698 INFO:tasks.workunit.client.0.smithi032.stdout:1.1c 2 0 2 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.291485+0000 34'44 75:103 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:49.691404+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:48.740642+0000 0 0 2024-09-13T12:11:56.698 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 3 0 3 0 0 1449 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.291469+0000 42'44 75:133 [0,2] 0 [0,2] 0 32'43 2024-09-13T12:09:43.710370+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:01:49.441566+0000 2 0 2024-09-13T12:11:56.698 INFO:tasks.workunit.client.0.smithi032.stdout:1.d 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422420+0000 42'45 76:122 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:50.728150+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:24:12.357955+0000 1 0 2024-09-13T12:11:56.698 INFO:tasks.workunit.client.0.smithi032.stdout:1.e 4 0 4 0 0 1603 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291323+0000 73'46 75:125 [0,2] 0 [0,2] 0 32'43 2024-09-13T12:09:49.907160+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:38:43.613716+0000 2 0 2024-09-13T12:11:56.698 INFO:tasks.workunit.client.0.smithi032.stdout:1.f 2 0 2 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.363578+0000 38'44 76:102 [2,0] 2 [2,0] 2 38'44 2024-09-13T12:09:59.616191+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:56.698 INFO:tasks.workunit.client.0.smithi032.stdout:1.10 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.291344+0000 43'45 75:137 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:52.759350+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T12:46:25.509739+0000 2 0 2024-09-13T12:11:56.699 INFO:tasks.workunit.client.0.smithi032.stdout:1.11 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.363401+0000 34'45 76:129 [2,0] 2 [2,0] 2 34'45 2024-09-13T12:09:53.421579+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T15:04:47.312019+0000 3 0 2024-09-13T12:11:56.699 INFO:tasks.workunit.client.0.smithi032.stdout:1.12 1 0 1 0 0 483 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.290912+0000 34'44 75:127 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:52.858658+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T23:50:26.431309+0000 1 0 2024-09-13T12:11:56.699 INFO:tasks.workunit.client.0.smithi032.stdout:1.13 3 0 3 0 0 1615 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.939197+0000 44'45 76:125 [2,0] 2 [2,0] 2 44'45 2024-09-13T12:10:04.717933+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:55.610436+0000 0 0 2024-09-13T12:11:56.699 INFO:tasks.workunit.client.0.smithi032.stdout:1.14 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.291845+0000 44'45 75:139 [0,2] 0 [0,2] 0 32'43 2024-09-13T12:09:50.727967+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T18:54:37.906181+0000 1 0 2024-09-13T12:11:56.699 INFO:tasks.workunit.client.0.smithi032.stdout:1.15 3 0 3 0 0 1449 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422194+0000 38'44 76:117 [2,0] 2 [2,0] 2 38'44 2024-09-13T12:09:54.402988+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:26:56.754038+0000 3 0 2024-09-13T12:11:56.699 INFO:tasks.workunit.client.0.smithi032.stdout:1.16 4 0 4 0 0 1932 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291960+0000 38'46 75:142 [0,2] 0 [0,2] 0 38'46 2024-09-13T12:10:00.648885+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:11:39.586367+0000 4 0 2024-09-13T12:11:56.699 INFO:tasks.workunit.client.0.smithi032.stdout:1.17 1 0 1 0 0 649 0 0 13 30 13 active+undersized+degraded 2024-09-13T12:10:32.423246+0000 32'43 76:101 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:56.659002+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:56.699 INFO:tasks.workunit.client.0.smithi032.stdout:1.18 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422567+0000 44'45 76:132 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:47.765518+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T12:54:37.159964+0000 1 0 2024-09-13T12:11:56.699 INFO:tasks.workunit.client.0.smithi032.stdout:1.19 2 0 2 0 0 966 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422893+0000 34'44 76:144 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:51.375338+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:16:57.418519+0000 2 0 2024-09-13T12:11:56.699 INFO:tasks.workunit.client.0.smithi032.stdout:1.1a 4 0 4 0 0 1932 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.423351+0000 41'45 76:124 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:51.953415+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T21:57:52.925654+0000 3 0 2024-09-13T12:11:56.699 INFO:tasks.workunit.client.0.smithi032.stdout:1.1b 5 0 5 0 0 2415 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291329+0000 44'46 75:144 [0,2] 0 [0,2] 0 36'45 2024-09-13T12:09:54.677751+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T15:59:12.687663+0000 4 0 2024-09-13T12:11:56.699 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:11:56.699 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:11:56.700 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:11:56.708 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: rados -p test put foo /etc/fstab 2024-09-13T12:11:56.817 INFO:tasks.workunit.client.0.smithi032.stdout:killing divergent 1 2024-09-13T12:11:56.817 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: echo 'killing divergent 1' 2024-09-13T12:11:56.817 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:793: TEST_divergent_3: ceph pg dump pgs 2024-09-13T12:11:57.207 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:11:57.208 INFO:tasks.workunit.client.0.smithi032.stdout:1.1f 3 0 3 0 0 1615 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422575+0000 38'45 76:114 [2,0] 2 [2,0] 2 38'45 2024-09-13T12:09:53.714380+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:56.617901+0000 0 0 2024-09-13T12:11:57.208 INFO:tasks.workunit.client.0.smithi032.stdout:1.1e 3 0 3 0 0 1615 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.291074+0000 44'45 75:121 [0,2] 0 [0,2] 0 44'45 2024-09-13T12:10:01.695428+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:50.801987+0000 0 0 2024-09-13T12:11:57.208 INFO:tasks.workunit.client.0.smithi032.stdout:1.1d 4 0 4 0 0 1932 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:32.422167+0000 39'46 76:126 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:49.397412+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T15:29:37.811752+0000 2 0 2024-09-13T12:11:57.208 INFO:tasks.workunit.client.0.smithi032.stdout:1.b 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.290838+0000 42'44 75:121 [0,2] 0 [0,2] 0 42'44 2024-09-13T12:09:57.774856+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:51.792644+0000 0 0 2024-09-13T12:11:57.209 INFO:tasks.workunit.client.0.smithi032.stdout:1.a 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422529+0000 34'44 76:97 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:50.853058+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:54.598530+0000 0 0 2024-09-13T12:11:57.209 INFO:tasks.workunit.client.0.smithi032.stdout:1.9 5 0 5 0 0 2415 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.036386+0000 42'45 76:137 [2,0] 2 [2,0] 2 42'45 2024-09-13T12:10:03.691408+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T17:45:19.630728+0000 4 0 2024-09-13T12:11:57.209 INFO:tasks.workunit.client.0.smithi032.stdout:1.8 5 0 5 0 0 2098 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:32.422809+0000 42'46 76:120 [2,0] 2 [2,0] 2 42'46 2024-09-13T12:09:58.652148+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:57.589433+0000 0 0 2024-09-13T12:11:57.209 INFO:tasks.workunit.client.0.smithi032.stdout:1.7 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422184+0000 34'44 76:114 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:55.719570+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:53.645748+0000 0 0 2024-09-13T12:11:57.209 INFO:tasks.workunit.client.0.smithi032.stdout:1.6 5 0 5 0 0 2098 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291701+0000 41'46 75:134 [0,2] 0 [0,2] 0 41'46 2024-09-13T12:10:02.648875+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:49.769629+0000 0 0 2024-09-13T12:11:57.210 INFO:tasks.workunit.client.0.smithi032.stdout:1.5 6 0 6 0 0 2898 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:32.422143+0000 42'46 76:123 [2,0] 2 [2,0] 2 38'45 2024-09-13T12:09:52.375427+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T21:28:19.785529+0000 5 0 2024-09-13T12:11:57.210 INFO:tasks.workunit.client.0.smithi032.stdout:1.4 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.290860+0000 34'44 75:104 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:46.701189+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:57.210 INFO:tasks.workunit.client.0.smithi032.stdout:1.3 5 0 5 0 0 2415 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.363328+0000 44'45 76:128 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:44.638103+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T20:31:20.891228+0000 3 0 2024-09-13T12:11:57.210 INFO:tasks.workunit.client.0.smithi032.stdout:1.2 1 0 1 0 0 483 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.291679+0000 39'44 75:122 [0,2] 0 [0,2] 0 39'44 2024-09-13T12:09:55.946588+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T18:12:36.231144+0000 1 0 2024-09-13T12:11:57.210 INFO:tasks.workunit.client.0.smithi032.stdout:1.c 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.290976+0000 41'45 75:131 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:51.776896+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:19:05.496290+0000 2 0 2024-09-13T12:11:57.210 INFO:tasks.workunit.client.0.smithi032.stdout:1.1 2 0 2 0 0 966 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422961+0000 41'45 76:119 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:48.455862+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:18:33.965292+0000 0 0 2024-09-13T12:11:57.211 INFO:tasks.workunit.client.0.smithi032.stdout:1.1c 2 0 2 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.291485+0000 34'44 75:103 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:49.691404+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:48.740642+0000 0 0 2024-09-13T12:11:57.211 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 3 0 3 0 0 1449 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.291469+0000 42'44 75:133 [0,2] 0 [0,2] 0 32'43 2024-09-13T12:09:43.710370+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:01:49.441566+0000 2 0 2024-09-13T12:11:57.211 INFO:tasks.workunit.client.0.smithi032.stdout:1.d 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422420+0000 42'45 76:122 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:50.728150+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:24:12.357955+0000 1 0 2024-09-13T12:11:57.211 INFO:tasks.workunit.client.0.smithi032.stdout:1.e 4 0 4 0 0 1603 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291323+0000 73'46 75:125 [0,2] 0 [0,2] 0 32'43 2024-09-13T12:09:49.907160+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:38:43.613716+0000 2 0 2024-09-13T12:11:57.211 INFO:tasks.workunit.client.0.smithi032.stdout:1.f 2 0 2 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.363578+0000 38'44 76:102 [2,0] 2 [2,0] 2 38'44 2024-09-13T12:09:59.616191+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:57.212 INFO:tasks.workunit.client.0.smithi032.stdout:1.10 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.291344+0000 43'45 75:137 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:52.759350+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T12:46:25.509739+0000 2 0 2024-09-13T12:11:57.212 INFO:tasks.workunit.client.0.smithi032.stdout:1.11 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.363401+0000 34'45 76:129 [2,0] 2 [2,0] 2 34'45 2024-09-13T12:09:53.421579+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T15:04:47.312019+0000 3 0 2024-09-13T12:11:57.212 INFO:tasks.workunit.client.0.smithi032.stdout:1.12 1 0 1 0 0 483 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.290912+0000 34'44 75:127 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:52.858658+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T23:50:26.431309+0000 1 0 2024-09-13T12:11:57.212 INFO:tasks.workunit.client.0.smithi032.stdout:1.13 3 0 3 0 0 1615 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.939197+0000 44'45 76:125 [2,0] 2 [2,0] 2 44'45 2024-09-13T12:10:04.717933+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:55.610436+0000 0 0 2024-09-13T12:11:57.212 INFO:tasks.workunit.client.0.smithi032.stdout:1.14 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.291845+0000 44'45 75:139 [0,2] 0 [0,2] 0 32'43 2024-09-13T12:09:50.727967+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T18:54:37.906181+0000 1 0 2024-09-13T12:11:57.213 INFO:tasks.workunit.client.0.smithi032.stdout:1.15 3 0 3 0 0 1449 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422194+0000 38'44 76:117 [2,0] 2 [2,0] 2 38'44 2024-09-13T12:09:54.402988+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:26:56.754038+0000 3 0 2024-09-13T12:11:57.213 INFO:tasks.workunit.client.0.smithi032.stdout:1.16 4 0 4 0 0 1932 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291960+0000 38'46 75:142 [0,2] 0 [0,2] 0 38'46 2024-09-13T12:10:00.648885+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:11:39.586367+0000 4 0 2024-09-13T12:11:57.213 INFO:tasks.workunit.client.0.smithi032.stdout:1.17 1 0 1 0 0 649 0 0 13 30 13 active+undersized+degraded 2024-09-13T12:10:32.423246+0000 32'43 76:101 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:56.659002+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:11:57.213 INFO:tasks.workunit.client.0.smithi032.stdout:1.18 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422567+0000 44'45 76:132 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:47.765518+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T12:54:37.159964+0000 1 0 2024-09-13T12:11:57.213 INFO:tasks.workunit.client.0.smithi032.stdout:1.19 2 0 2 0 0 966 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422893+0000 34'44 76:144 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:51.375338+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:16:57.418519+0000 2 0 2024-09-13T12:11:57.214 INFO:tasks.workunit.client.0.smithi032.stdout:1.1a 4 0 4 0 0 1932 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.423351+0000 41'45 76:124 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:51.953415+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T21:57:52.925654+0000 3 0 2024-09-13T12:11:57.214 INFO:tasks.workunit.client.0.smithi032.stdout:1.1b 5 0 5 0 0 2415 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291329+0000 44'46 75:144 [0,2] 0 [0,2] 0 36'45 2024-09-13T12:09:54.677751+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T15:59:12.687663+0000 4 0 2024-09-13T12:11:57.214 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:11:57.214 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:11:57.214 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:11:57.218 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.1 2024-09-13T12:11:57.219 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:11:57.219 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:11:57.219 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:11:57.220 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:11:57.220 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:11:57.325 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:11:57.325 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: expfile=td/divergent-priors/exp.33242.out 2024-09-13T12:11:57.325 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:798: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.33242.out 2024-09-13T12:11:57.326 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-13T12:11:57.326 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T12:11:57.326 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T12:11:57.326 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T12:11:57.326 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-13T12:11:57.326 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.33242.out 2024-09-13T12:11:58.688 INFO:tasks.workunit.client.0.smithi032.stderr:Exporting 1.1b info 1.1b( v 44'46 lc 0'0 (31'30,44'46] local-lis/les=78/79 n=5 ec=33/23 lis/c=78/52 les/c/f=79/53/0 sis=78) 2024-09-13T12:11:58.688 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-09-13T12:11:58.688 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-09-13T12:11:58.688 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:dd677bd3:::existing_35:head# 2024-09-13T12:11:58.688 INFO:tasks.workunit.client.0.smithi032.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-09-13T12:11:58.689 INFO:tasks.workunit.client.0.smithi032.stderr:Export successful 2024-09-13T12:11:58.689 INFO:tasks.workunit.client.0.smithi032.stdout: marking collection for removal 2024-09-13T12:11:58.690 INFO:tasks.workunit.client.0.smithi032.stdout:setting '_remove' omap key 2024-09-13T12:11:58.690 INFO:tasks.workunit.client.0.smithi032.stdout:finish_remove_pgs 1.1b_head removing 1.1b 2024-09-13T12:11:58.794 INFO:tasks.workunit.client.0.smithi032.stderr:Remove successful 2024-09-13T12:11:59.203 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.33242.out 2024-09-13T12:11:59.203 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-13T12:11:59.203 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T12:11:59.203 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T12:11:59.203 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T12:11:59.204 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-13T12:11:59.204 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.33242.out 2024-09-13T12:12:00.332 INFO:tasks.workunit.client.0.smithi032.stdout:get_pg_num_history pg_num_history pg_num_history(e80 pg_nums {1={23=1,33=32}} deleted_pools ) 2024-09-13T12:12:00.333 INFO:tasks.workunit.client.0.smithi032.stdout:Importing pgid 1.1b 2024-09-13T12:12:00.333 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-09-13T12:12:00.333 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:12:00.406 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-09-13T12:12:00.406 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:12:00.440 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:dd677bd3:::existing_35:head# 2024-09-13T12:12:00.440 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:12:00.482 INFO:tasks.workunit.client.0.smithi032.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-09-13T12:12:00.482 INFO:tasks.workunit.client.0.smithi032.stdout:snapset 0=[]:{} 2024-09-13T12:12:00.531 INFO:tasks.workunit.client.0.smithi032.stdout:write_pg epoch 79 info 1.1b( v 44'46 lc 0'0 (31'30,44'46] local-lis/les=78/79 n=5 ec=33/23 lis/c=78/52 les/c/f=79/53/0 sis=78) 2024-09-13T12:12:00.590 INFO:tasks.workunit.client.0.smithi032.stdout:Import successful 2024-09-13T12:12:01.122 INFO:tasks.workunit.client.0.smithi032.stdout:reviving divergent 1 2024-09-13T12:12:01.122 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:801: TEST_divergent_3: echo 'reviving divergent 1' 2024-09-13T12:12:01.122 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: ceph pg dump pgs 2024-09-13T12:12:01.528 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:12:01.528 INFO:tasks.workunit.client.0.smithi032.stdout:1.1f 3 0 3 0 0 1615 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422575+0000 38'45 76:114 [2,0] 2 [2,0] 2 38'45 2024-09-13T12:09:53.714380+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:56.617901+0000 0 0 2024-09-13T12:12:01.528 INFO:tasks.workunit.client.0.smithi032.stdout:1.1e 3 0 3 0 0 1615 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.291074+0000 44'45 75:121 [0,2] 0 [0,2] 0 44'45 2024-09-13T12:10:01.695428+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:50.801987+0000 0 0 2024-09-13T12:12:01.528 INFO:tasks.workunit.client.0.smithi032.stdout:1.1d 4 0 0 0 0 1932 0 0 16 30 16 active+clean 2024-09-13T12:11:55.520292+0000 39'46 80:139 [2,1,0] 2 [2,1,0] 2 34'44 2024-09-13T12:09:49.397412+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T17:24:59.534501+0000 2 0 2024-09-13T12:12:01.528 INFO:tasks.workunit.client.0.smithi032.stdout:1.b 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.290838+0000 42'44 75:121 [0,2] 0 [0,2] 0 42'44 2024-09-13T12:09:57.774856+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:51.792644+0000 0 0 2024-09-13T12:12:01.528 INFO:tasks.workunit.client.0.smithi032.stdout:1.a 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422529+0000 34'44 76:97 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:50.853058+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:54.598530+0000 0 0 2024-09-13T12:12:01.528 INFO:tasks.workunit.client.0.smithi032.stdout:1.9 5 0 5 0 0 2415 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.036386+0000 42'45 76:137 [2,0] 2 [2,0] 2 42'45 2024-09-13T12:10:03.691408+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T17:45:19.630728+0000 4 0 2024-09-13T12:12:01.529 INFO:tasks.workunit.client.0.smithi032.stdout:1.8 5 0 5 0 0 2098 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:32.422809+0000 42'46 76:120 [2,0] 2 [2,0] 2 42'46 2024-09-13T12:09:58.652148+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:57.589433+0000 0 0 2024-09-13T12:12:01.529 INFO:tasks.workunit.client.0.smithi032.stdout:1.7 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422184+0000 34'44 76:114 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:55.719570+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:53.645748+0000 0 0 2024-09-13T12:12:01.529 INFO:tasks.workunit.client.0.smithi032.stdout:1.6 5 0 5 0 0 2098 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291701+0000 41'46 75:134 [0,2] 0 [0,2] 0 41'46 2024-09-13T12:10:02.648875+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:49.769629+0000 0 0 2024-09-13T12:12:01.529 INFO:tasks.workunit.client.0.smithi032.stdout:1.5 6 0 0 0 0 2898 0 0 16 30 16 active+clean 2024-09-13T12:11:55.520082+0000 42'46 80:136 [2,0,1] 2 [2,0,1] 2 38'45 2024-09-13T12:09:52.375427+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T18:45:43.756648+0000 5 0 2024-09-13T12:12:01.529 INFO:tasks.workunit.client.0.smithi032.stdout:1.4 3 0 3 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.290860+0000 34'44 75:104 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:46.701189+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:12:01.529 INFO:tasks.workunit.client.0.smithi032.stdout:1.3 5 0 5 0 0 2415 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.363328+0000 44'45 76:128 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:44.638103+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T20:31:20.891228+0000 3 0 2024-09-13T12:12:01.529 INFO:tasks.workunit.client.0.smithi032.stdout:1.2 1 0 0 0 0 483 0 0 14 30 14 active+clean 2024-09-13T12:11:55.024953+0000 39'44 80:139 [0,1,2] 0 [0,1,2] 0 39'44 2024-09-13T12:09:55.946588+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T21:45:33.899662+0000 1 0 2024-09-13T12:12:01.530 INFO:tasks.workunit.client.0.smithi032.stdout:1.c 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.290976+0000 41'45 75:131 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:51.776896+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:19:05.496290+0000 2 0 2024-09-13T12:12:01.530 INFO:tasks.workunit.client.0.smithi032.stdout:1.1 2 0 0 0 0 966 0 0 15 30 15 active+clean 2024-09-13T12:11:55.519672+0000 41'45 80:132 [2,0,1] 2 [2,0,1] 2 32'43 2024-09-13T12:09:48.455862+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T21:23:20.656341+0000 0 0 2024-09-13T12:12:01.530 INFO:tasks.workunit.client.0.smithi032.stdout:1.1c 2 0 2 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.291485+0000 34'44 75:103 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:49.691404+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:48.740642+0000 0 0 2024-09-13T12:12:01.530 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 3 0 3 0 0 1449 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:31.291469+0000 42'44 75:133 [0,2] 0 [0,2] 0 32'43 2024-09-13T12:09:43.710370+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:01:49.441566+0000 2 0 2024-09-13T12:12:01.530 INFO:tasks.workunit.client.0.smithi032.stdout:1.d 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:11:55.519097+0000 42'45 80:135 [2,1,0] 2 [2,1,0] 2 32'43 2024-09-13T12:09:50.728150+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T22:13:09.777587+0000 1 0 2024-09-13T12:12:01.530 INFO:tasks.workunit.client.0.smithi032.stdout:1.e 4 0 4 0 0 1603 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291323+0000 73'46 75:125 [0,2] 0 [0,2] 0 32'43 2024-09-13T12:09:49.907160+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:38:43.613716+0000 2 0 2024-09-13T12:12:01.530 INFO:tasks.workunit.client.0.smithi032.stdout:1.f 2 0 2 0 0 1132 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.363578+0000 38'44 76:102 [2,0] 2 [2,0] 2 38'44 2024-09-13T12:09:59.616191+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:12:01.530 INFO:tasks.workunit.client.0.smithi032.stdout:1.10 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.291344+0000 43'45 75:137 [0,2] 0 [0,2] 0 34'44 2024-09-13T12:09:52.759350+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T12:46:25.509739+0000 2 0 2024-09-13T12:12:01.530 INFO:tasks.workunit.client.0.smithi032.stdout:1.11 3 0 1 0 0 1449 0 0 15 30 15 active+recovery_wait+degraded 2024-09-13T12:11:55.519038+0000 34'45 80:140 [2,0,1] 2 [2,0,1] 2 34'45 2024-09-13T12:09:53.421579+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 no scrub is scheduled 3 0 2024-09-13T12:12:01.530 INFO:tasks.workunit.client.0.smithi032.stdout:1.12 1 0 0 0 0 483 0 0 14 30 14 active+clean 2024-09-13T12:11:55.025282+0000 34'44 80:144 [0,1,2] 0 [0,1,2] 0 34'44 2024-09-13T12:09:52.858658+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T23:00:09.449201+0000 1 0 2024-09-13T12:12:01.531 INFO:tasks.workunit.client.0.smithi032.stdout:1.13 3 0 3 0 0 1615 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.939197+0000 44'45 76:125 [2,0] 2 [2,0] 2 44'45 2024-09-13T12:10:04.717933+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 user requested scrub scheduled @ 2024-09-13T12:11:55.610436+0000 0 0 2024-09-13T12:12:01.531 INFO:tasks.workunit.client.0.smithi032.stdout:1.14 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:31.291845+0000 44'45 75:139 [0,2] 0 [0,2] 0 32'43 2024-09-13T12:09:50.727967+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T18:54:37.906181+0000 1 0 2024-09-13T12:12:01.531 INFO:tasks.workunit.client.0.smithi032.stdout:1.15 3 0 0 0 0 1449 0 0 14 30 14 active+clean 2024-09-13T12:11:55.519382+0000 38'44 80:130 [2,0,1] 2 [2,0,1] 2 38'44 2024-09-13T12:09:54.402988+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:25:14.609181+0000 3 0 2024-09-13T12:12:01.531 INFO:tasks.workunit.client.0.smithi032.stdout:1.16 4 0 4 0 0 1932 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291960+0000 38'46 75:142 [0,2] 0 [0,2] 0 38'46 2024-09-13T12:10:00.648885+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:11:39.586367+0000 4 0 2024-09-13T12:12:01.531 INFO:tasks.workunit.client.0.smithi032.stdout:1.17 1 0 1 0 0 649 0 0 13 30 13 active+undersized+degraded 2024-09-13T12:10:32.423246+0000 32'43 76:101 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:56.659002+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 queued for scrub 0 0 2024-09-13T12:12:01.531 INFO:tasks.workunit.client.0.smithi032.stdout:1.18 3 0 3 0 0 1449 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.422567+0000 44'45 76:132 [2,0] 2 [2,0] 2 32'43 2024-09-13T12:09:47.765518+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T12:54:37.159964+0000 1 0 2024-09-13T12:12:01.531 INFO:tasks.workunit.client.0.smithi032.stdout:1.19 2 0 2 0 0 966 0 0 14 30 14 active+undersized+degraded 2024-09-13T12:10:32.422893+0000 34'44 76:144 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:51.375338+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:16:57.418519+0000 2 0 2024-09-13T12:12:01.532 INFO:tasks.workunit.client.0.smithi032.stdout:1.1a 4 0 4 0 0 1932 0 0 15 30 15 active+undersized+degraded 2024-09-13T12:10:32.423351+0000 41'45 76:124 [2,0] 2 [2,0] 2 34'44 2024-09-13T12:09:51.953415+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T21:57:52.925654+0000 3 0 2024-09-13T12:12:01.532 INFO:tasks.workunit.client.0.smithi032.stdout:1.1b 5 0 5 0 0 2415 0 0 16 30 16 active+undersized+degraded 2024-09-13T12:10:31.291329+0000 44'46 75:144 [0,2] 0 [0,2] 0 36'45 2024-09-13T12:09:54.677751+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T15:59:12.687663+0000 4 0 2024-09-13T12:12:01.532 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:12:01.532 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:12:01.532 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:12:01.542 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: activate_osd td/divergent-priors 1 2024-09-13T12:12:01.543 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-13T12:12:01.543 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:12:01.543 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T12:12:01.543 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:12:01.543 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-13T12:12:01.543 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T12:12:01.543 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:12:01.543 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:12:01.543 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:12:01.543 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-13T12:12:01.543 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-13T12:12:01.543 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:12:01.544 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:12:01.544 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T12:12:01.544 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:12:01.544 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:12:01.544 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:12:01.544 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:12:01.544 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:12:01.544 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:12:01.545 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:12:01.545 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:12:01.545 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:12:01.545 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T12:12:01.545 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T12:12:01.545 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:12:01.545 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:12:01.545 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:12:01.545 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:12:01.545 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:12:01.545 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:12:01.545 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-13T12:12:01.546 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.1 2024-09-13T12:12:01.546 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T12:12:01.547 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:12:01.547 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-13T12:12:01.547 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T12:12:01.547 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:12:01.547 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:12:01.547 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:12:01.577 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:12:01.576+0000 7fc7ac3d4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:12:01.579 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:12:01.578+0000 7fc7ac3d4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:12:01.580 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:12:01.579+0000 7fc7ac3d4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:12:01.981 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T12:12:01.981 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:12:01.981 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T12:12:01.981 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:12:01.981 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:12:01.981 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:12:01.982 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:12:01.982 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:12:01.982 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:12:01.982 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:12:02.414 INFO:tasks.workunit.client.0.smithi032.stdout:osd.1 up in weight 1 up_from 78 up_thru 78 down_at 58 last_clean_interval [10,56) [v2:127.0.0.1:6818/108162265,v1:127.0.0.1:6819/108162265] [v2:127.0.0.1:6820/108162265,v1:127.0.0.1:6821/108162265] exists,up f377d2cc-d182-4459-bd79-b23c9379a371 2024-09-13T12:12:02.415 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:12:02.415 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:12:02.415 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:12:02.415 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: wait_for_osd up 1 2024-09-13T12:12:02.415 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:12:02.416 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T12:12:02.416 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:12:02.416 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:12:02.416 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:12:02.416 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:12:02.416 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:12:02.417 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:12:02.417 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:12:02.857 INFO:tasks.workunit.client.0.smithi032.stdout:osd.1 up in weight 1 up_from 78 up_thru 78 down_at 58 last_clean_interval [10,56) [v2:127.0.0.1:6818/108162265,v1:127.0.0.1:6819/108162265] [v2:127.0.0.1:6820/108162265,v1:127.0.0.1:6821/108162265] exists,up f377d2cc-d182-4459-bd79-b23c9379a371 2024-09-13T12:12:02.858 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:12:02.858 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:12:02.858 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:12:02.858 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: sleep 20 2024-09-13T12:12:03.183 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:12:03.183+0000 7fc7ac3d4640 -1 Falling back to public interface 2024-09-13T12:12:06.051 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:12:06.049+0000 7fc7ac3d4640 -1 osd.1 80 log_to_monitors true 2024-09-13T12:12:22.861 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: get_asok_path osd.1 2024-09-13T12:12:22.862 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-13T12:12:22.862 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-13T12:12:22.862 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:12:22.862 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:12:22.862 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:12:22.863 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-osd.1.asok 2024-09-13T12:12:22.863 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: CEPH_ARGS= 2024-09-13T12:12:22.863 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.33242/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T12:12:22.952 INFO:tasks.workunit.client.0.smithi032.stdout:{ 2024-09-13T12:12:22.952 INFO:tasks.workunit.client.0.smithi032.stdout: "ops": [], 2024-09-13T12:12:22.952 INFO:tasks.workunit.client.0.smithi032.stdout: "num_ops": 0 2024-09-13T12:12:22.952 INFO:tasks.workunit.client.0.smithi032.stdout:} 2024-09-13T12:12:22.959 INFO:tasks.workunit.client.0.smithi032.stdout:allowing recovery 2024-09-13T12:12:22.960 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:809: TEST_divergent_3: echo 'allowing recovery' 2024-09-13T12:12:22.960 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: ceph pg dump pgs 2024-09-13T12:12:23.351 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:12:23.351 INFO:tasks.workunit.client.0.smithi032.stdout:1.1f 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:12:07.843048+0000 38'45 85:153 [1,2,0] 1 [1,2,0] 1 38'45 2024-09-13T12:11:56.271975+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:33:59.286826+0000 3 0 2024-09-13T12:12:23.351 INFO:tasks.workunit.client.0.smithi032.stdout:1.1e 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:12:16.208538+0000 44'45 85:150 [1,0,2] 1 [1,0,2] 1 44'45 2024-09-13T12:12:16.208443+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T18:07:01.345183+0000 3 0 2024-09-13T12:12:23.351 INFO:tasks.workunit.client.0.smithi032.stdout:1.1d 4 0 0 0 0 1932 0 0 16 30 16 active+clean 2024-09-13T12:12:07.835456+0000 39'46 85:153 [2,1,0] 2 [2,1,0] 2 34'44 2024-09-13T12:09:49.397412+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:47:56.617222+0000 2 0 2024-09-13T12:12:23.351 INFO:tasks.workunit.client.0.smithi032.stdout:1.b 3 0 0 0 0 1449 0 0 14 30 14 active+clean 2024-09-13T12:12:12.171146+0000 42'44 85:177 [1,0,2] 1 [1,0,2] 1 42'44 2024-09-13T12:12:12.171083+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T17:19:32.114565+0000 3 0 2024-09-13T12:12:23.351 INFO:tasks.workunit.client.0.smithi032.stdout:1.a 3 0 0 0 0 1449 0 0 14 30 14 active+clean 2024-09-13T12:12:07.844741+0000 34'44 85:136 [1,2,0] 1 [1,2,0] 1 34'44 2024-09-13T12:10:06.700135+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:41:03.180227+0000 3 0 2024-09-13T12:12:23.351 INFO:tasks.workunit.client.0.smithi032.stdout:1.9 5 1 1 0 0 2415 0 0 15 30 15 active+recovery_wait+degraded 2024-09-13T12:12:07.837061+0000 42'45 85:153 [1,2,0] 1 [1,2,0] 1 42'45 2024-09-13T12:10:03.691408+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 no scrub is scheduled 4 0 2024-09-13T12:12:23.351 INFO:tasks.workunit.client.0.smithi032.stdout:1.8 3 0 0 0 0 1449 0 0 16 30 16 active+clean 2024-09-13T12:12:11.120778+0000 42'46 85:168 [1,2,0] 1 [1,2,0] 1 42'46 2024-09-13T12:12:11.120705+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T23:34:48.468455+0000 3 0 2024-09-13T12:12:23.351 INFO:tasks.workunit.client.0.smithi032.stdout:1.7 2 0 0 0 0 966 0 0 14 30 14 active+clean 2024-09-13T12:12:14.201864+0000 34'44 85:142 [1,2,0] 1 [1,2,0] 1 34'44 2024-09-13T12:12:14.201800+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:37:31.180061+0000 2 0 2024-09-13T12:12:23.351 INFO:tasks.workunit.client.0.smithi032.stdout:1.6 5 0 0 0 0 2415 0 0 17 30 17 active+clean 2024-09-13T12:12:07.843428+0000 80'47 85:171 [1,0,2] 1 [1,0,2] 1 41'46 2024-09-13T12:11:55.335844+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T22:07:35.302154+0000 4 0 2024-09-13T12:12:23.352 INFO:tasks.workunit.client.0.smithi032.stdout:1.5 6 0 0 0 0 2898 0 0 16 30 16 active+clean 2024-09-13T12:12:07.836232+0000 42'46 85:150 [2,0,1] 2 [2,0,1] 2 38'45 2024-09-13T12:09:52.375427+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T21:30:34.945529+0000 5 0 2024-09-13T12:12:23.352 INFO:tasks.workunit.client.0.smithi032.stdout:1.4 2 0 0 0 0 966 0 0 14 30 14 active+clean 2024-09-13T12:12:13.179604+0000 34'44 85:125 [1,0,2] 1 [1,0,2] 1 34'44 2024-09-13T12:12:13.179535+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T18:47:06.781447+0000 2 0 2024-09-13T12:12:23.352 INFO:tasks.workunit.client.0.smithi032.stdout:1.3 5 0 0 0 0 2415 0 0 15 30 15 active+clean 2024-09-13T12:12:07.839409+0000 44'45 85:141 [1,2,0] 1 [1,2,0] 1 32'43 2024-09-13T12:09:44.638103+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T18:25:27.318335+0000 3 0 2024-09-13T12:12:23.352 INFO:tasks.workunit.client.0.smithi032.stdout:1.2 1 0 0 0 0 483 0 0 14 30 14 active+clean 2024-09-13T12:12:07.834856+0000 39'44 85:154 [0,1,2] 0 [0,1,2] 0 39'44 2024-09-13T12:09:55.946588+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T17:08:45.051178+0000 1 0 2024-09-13T12:12:23.352 INFO:tasks.workunit.client.0.smithi032.stdout:1.c 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:12:07.843071+0000 41'45 85:137 [1,0,2] 1 [1,0,2] 1 34'44 2024-09-13T12:09:51.776896+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T14:32:03.924204+0000 2 0 2024-09-13T12:12:23.352 INFO:tasks.workunit.client.0.smithi032.stdout:1.1 2 0 0 0 0 966 0 0 15 30 15 active+clean 2024-09-13T12:12:07.837058+0000 41'45 85:146 [2,0,1] 2 [2,0,1] 2 32'43 2024-09-13T12:09:48.455862+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T12:15:05.886327+0000 0 0 2024-09-13T12:12:23.352 INFO:tasks.workunit.client.0.smithi032.stdout:1.1c 2 0 0 0 0 966 0 0 14 30 14 active+clean 2024-09-13T12:12:08.077789+0000 34'44 85:135 [1,0,2] 1 [1,0,2] 1 34'44 2024-09-13T12:12:08.077728+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T23:35:35.627097+0000 2 0 2024-09-13T12:12:23.352 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 3 0 0 0 0 1449 0 0 14 30 14 active+clean 2024-09-13T12:12:07.845648+0000 42'44 85:139 [1,0,2] 1 [1,0,2] 1 32'43 2024-09-13T12:09:43.710370+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:54:34.880745+0000 2 0 2024-09-13T12:12:23.352 INFO:tasks.workunit.client.0.smithi032.stdout:1.d 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:12:07.835640+0000 42'45 85:149 [2,1,0] 2 [2,1,0] 2 32'43 2024-09-13T12:09:50.728150+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T17:50:01.888601+0000 1 0 2024-09-13T12:12:23.352 INFO:tasks.workunit.client.0.smithi032.stdout:1.e 4 1 1 0 0 1603 0 0 16 30 16 active+recovery_wait+degraded 2024-09-13T12:12:07.841687+0000 73'46 85:153 [1,0,2] 1 [1,0,2] 1 32'43 2024-09-13T12:09:49.907160+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 no scrub is scheduled 2 0 2024-09-13T12:12:23.352 INFO:tasks.workunit.client.0.smithi032.stdout:1.f 4 0 0 0 0 1932 0 0 14 30 14 active+clean 2024-09-13T12:12:17.209549+0000 38'44 85:149 [1,2,0] 1 [1,2,0] 1 38'44 2024-09-13T12:12:17.209485+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T18:14:52.514325+0000 4 0 2024-09-13T12:12:23.352 INFO:tasks.workunit.client.0.smithi032.stdout:1.10 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:12:07.842621+0000 43'45 85:143 [1,0,2] 1 [1,0,2] 1 34'44 2024-09-13T12:09:52.759350+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:54:31.431312+0000 2 0 2024-09-13T12:12:23.353 INFO:tasks.workunit.client.0.smithi032.stdout:1.11 3 0 1 0 0 1449 0 0 15 30 15 active+recovering+degraded 2024-09-13T12:12:07.844295+0000 34'45 85:153 [2,0,1] 2 [2,0,1] 2 34'45 2024-09-13T12:09:53.421579+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 no scrub is scheduled 3 0 2024-09-13T12:12:23.353 INFO:tasks.workunit.client.0.smithi032.stdout:1.12 1 0 0 0 0 483 0 0 14 30 14 active+clean 2024-09-13T12:12:07.835667+0000 34'44 85:159 [0,1,2] 0 [0,1,2] 0 34'44 2024-09-13T12:09:52.858658+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T17:56:42.489529+0000 1 0 2024-09-13T12:12:23.353 INFO:tasks.workunit.client.0.smithi032.stdout:1.13 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:12:10.075207+0000 44'45 85:166 [1,2,0] 1 [1,2,0] 1 44'45 2024-09-13T12:12:10.075145+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:08:22.156011+0000 3 0 2024-09-13T12:12:23.353 INFO:tasks.workunit.client.0.smithi032.stdout:1.14 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:12:07.845969+0000 44'45 85:145 [1,0,2] 1 [1,0,2] 1 32'43 2024-09-13T12:09:50.727967+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:02:58.594723+0000 1 0 2024-09-13T12:12:23.353 INFO:tasks.workunit.client.0.smithi032.stdout:1.15 3 0 0 0 0 1449 0 0 14 30 14 active+clean 2024-09-13T12:12:07.837370+0000 38'44 85:144 [2,0,1] 2 [2,0,1] 2 38'44 2024-09-13T12:09:54.402988+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T15:25:25.878428+0000 3 0 2024-09-13T12:12:23.353 INFO:tasks.workunit.client.0.smithi032.stdout:1.16 4 0 0 0 0 1932 0 0 16 30 16 active+clean 2024-09-13T12:12:07.843705+0000 38'46 85:147 [1,0,2] 1 [1,0,2] 1 38'46 2024-09-13T12:10:00.648885+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T16:16:10.803318+0000 4 0 2024-09-13T12:12:23.353 INFO:tasks.workunit.client.0.smithi032.stdout:1.17 1 0 0 0 0 483 0 0 13 30 13 active+clean 2024-09-13T12:12:09.031942+0000 32'43 85:141 [1,2,0] 1 [1,2,0] 1 32'43 2024-09-13T12:12:09.031884+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T21:32:21.839825+0000 1 0 2024-09-13T12:12:23.353 INFO:tasks.workunit.client.0.smithi032.stdout:1.18 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:12:07.840908+0000 44'45 85:159 [1,2,0] 1 [1,2,0] 1 32'43 2024-09-13T12:09:47.765518+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T23:49:15.964977+0000 1 0 2024-09-13T12:12:23.353 INFO:tasks.workunit.client.0.smithi032.stdout:1.19 2 0 0 0 0 966 0 0 14 30 14 active+clean 2024-09-13T12:12:15.159508+0000 34'44 85:146 [1,2,0] 1 [1,2,0] 1 34'44 2024-09-13T12:12:15.159443+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T14:00:50.141731+0000 2 0 2024-09-13T12:12:23.353 INFO:tasks.workunit.client.0.smithi032.stdout:1.1a 4 0 0 0 0 1932 0 0 15 30 15 active+clean 2024-09-13T12:12:07.842209+0000 41'45 85:154 [1,2,0] 1 [1,2,0] 1 41'45 2024-09-13T12:10:05.687638+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:27:57.757227+0000 3 0 2024-09-13T12:12:23.353 INFO:tasks.workunit.client.0.smithi032.stdout:1.1b 5 1 1 0 0 2415 0 0 16 30 16 active+recovery_wait+degraded 2024-09-13T12:12:07.843335+0000 44'46 85:176 [1,0,2] 1 [1,0,2] 1 36'45 2024-09-13T12:09:54.677751+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 no scrub is scheduled 4 0 2024-09-13T12:12:23.353 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:12:23.354 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:12:23.354 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:12:23.362 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-09-13T12:12:23.362 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-13T12:12:23.478 INFO:tasks.workunit.client.0.smithi032.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-13T12:12:23.486 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-09-13T12:12:23.486 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-13T12:12:23.603 INFO:tasks.workunit.client.0.smithi032.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-13T12:12:23.612 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-09-13T12:12:23.612 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-13T12:12:23.727 INFO:tasks.workunit.client.0.smithi032.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-13T12:12:23.736 INFO:tasks.workunit.client.0.smithi032.stdout:reading divergent objects 2024-09-13T12:12:23.737 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:817: TEST_divergent_3: echo 'reading divergent objects' 2024-09-13T12:12:23.737 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:818: TEST_divergent_3: ceph pg dump pgs 2024-09-13T12:12:24.130 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:12:24.130 INFO:tasks.workunit.client.0.smithi032.stdout:1.1f 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:12:07.843048+0000 38'45 85:153 [1,2,0] 1 [1,2,0] 1 38'45 2024-09-13T12:11:56.271975+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:33:59.286826+0000 3 0 2024-09-13T12:12:24.130 INFO:tasks.workunit.client.0.smithi032.stdout:1.1e 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:12:16.208538+0000 44'45 85:150 [1,0,2] 1 [1,0,2] 1 44'45 2024-09-13T12:12:16.208443+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T18:07:01.345183+0000 3 0 2024-09-13T12:12:24.130 INFO:tasks.workunit.client.0.smithi032.stdout:1.1d 4 0 0 0 0 1932 0 0 16 30 16 active+clean 2024-09-13T12:12:07.835456+0000 39'46 85:153 [2,1,0] 2 [2,1,0] 2 34'44 2024-09-13T12:09:49.397412+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:47:56.617222+0000 2 0 2024-09-13T12:12:24.130 INFO:tasks.workunit.client.0.smithi032.stdout:1.b 3 0 0 0 0 1449 0 0 14 30 14 active+clean 2024-09-13T12:12:12.171146+0000 42'44 85:177 [1,0,2] 1 [1,0,2] 1 42'44 2024-09-13T12:12:12.171083+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T17:19:32.114565+0000 3 0 2024-09-13T12:12:24.130 INFO:tasks.workunit.client.0.smithi032.stdout:1.a 3 0 0 0 0 1449 0 0 14 30 14 active+clean 2024-09-13T12:12:07.844741+0000 34'44 85:136 [1,2,0] 1 [1,2,0] 1 34'44 2024-09-13T12:10:06.700135+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:41:03.180227+0000 3 0 2024-09-13T12:12:24.130 INFO:tasks.workunit.client.0.smithi032.stdout:1.9 5 1 1 0 0 2415 0 0 15 30 15 active+recovery_wait+degraded 2024-09-13T12:12:07.837061+0000 42'45 85:153 [1,2,0] 1 [1,2,0] 1 42'45 2024-09-13T12:10:03.691408+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 no scrub is scheduled 4 0 2024-09-13T12:12:24.130 INFO:tasks.workunit.client.0.smithi032.stdout:1.8 3 0 0 0 0 1449 0 0 16 30 16 active+clean 2024-09-13T12:12:11.120778+0000 42'46 85:168 [1,2,0] 1 [1,2,0] 1 42'46 2024-09-13T12:12:11.120705+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T23:34:48.468455+0000 3 0 2024-09-13T12:12:24.130 INFO:tasks.workunit.client.0.smithi032.stdout:1.7 2 0 0 0 0 966 0 0 14 30 14 active+clean 2024-09-13T12:12:14.201864+0000 34'44 85:142 [1,2,0] 1 [1,2,0] 1 34'44 2024-09-13T12:12:14.201800+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:37:31.180061+0000 2 0 2024-09-13T12:12:24.130 INFO:tasks.workunit.client.0.smithi032.stdout:1.6 5 0 0 0 0 2415 0 0 17 30 17 active+clean 2024-09-13T12:12:07.843428+0000 80'47 85:171 [1,0,2] 1 [1,0,2] 1 41'46 2024-09-13T12:11:55.335844+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T22:07:35.302154+0000 4 0 2024-09-13T12:12:24.131 INFO:tasks.workunit.client.0.smithi032.stdout:1.5 6 0 0 0 0 2898 0 0 16 30 16 active+clean 2024-09-13T12:12:07.836232+0000 42'46 85:150 [2,0,1] 2 [2,0,1] 2 38'45 2024-09-13T12:09:52.375427+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T21:30:34.945529+0000 5 0 2024-09-13T12:12:24.131 INFO:tasks.workunit.client.0.smithi032.stdout:1.4 2 0 0 0 0 966 0 0 14 30 14 active+clean 2024-09-13T12:12:13.179604+0000 34'44 85:125 [1,0,2] 1 [1,0,2] 1 34'44 2024-09-13T12:12:13.179535+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T18:47:06.781447+0000 2 0 2024-09-13T12:12:24.131 INFO:tasks.workunit.client.0.smithi032.stdout:1.3 5 0 0 0 0 2415 0 0 15 30 15 active+clean 2024-09-13T12:12:07.839409+0000 44'45 85:141 [1,2,0] 1 [1,2,0] 1 32'43 2024-09-13T12:09:44.638103+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T18:25:27.318335+0000 3 0 2024-09-13T12:12:24.131 INFO:tasks.workunit.client.0.smithi032.stdout:1.2 1 0 0 0 0 483 0 0 14 30 14 active+clean 2024-09-13T12:12:07.834856+0000 39'44 85:154 [0,1,2] 0 [0,1,2] 0 39'44 2024-09-13T12:09:55.946588+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T17:08:45.051178+0000 1 0 2024-09-13T12:12:24.131 INFO:tasks.workunit.client.0.smithi032.stdout:1.c 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:12:07.843071+0000 41'45 85:137 [1,0,2] 1 [1,0,2] 1 34'44 2024-09-13T12:09:51.776896+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T14:32:03.924204+0000 2 0 2024-09-13T12:12:24.131 INFO:tasks.workunit.client.0.smithi032.stdout:1.1 2 0 0 0 0 966 0 0 15 30 15 active+clean 2024-09-13T12:12:07.837058+0000 41'45 85:146 [2,0,1] 2 [2,0,1] 2 32'43 2024-09-13T12:09:48.455862+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T12:15:05.886327+0000 0 0 2024-09-13T12:12:24.131 INFO:tasks.workunit.client.0.smithi032.stdout:1.1c 2 0 0 0 0 966 0 0 14 30 14 active+clean 2024-09-13T12:12:08.077789+0000 34'44 85:135 [1,0,2] 1 [1,0,2] 1 34'44 2024-09-13T12:12:08.077728+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T23:35:35.627097+0000 2 0 2024-09-13T12:12:24.131 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 3 0 0 0 0 1449 0 0 14 30 14 active+clean 2024-09-13T12:12:07.845648+0000 42'44 85:139 [1,0,2] 1 [1,0,2] 1 32'43 2024-09-13T12:09:43.710370+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T19:54:34.880745+0000 2 0 2024-09-13T12:12:24.131 INFO:tasks.workunit.client.0.smithi032.stdout:1.d 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:12:07.835640+0000 42'45 85:149 [2,1,0] 2 [2,1,0] 2 32'43 2024-09-13T12:09:50.728150+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T17:50:01.888601+0000 1 0 2024-09-13T12:12:24.131 INFO:tasks.workunit.client.0.smithi032.stdout:1.e 4 1 1 0 0 1603 0 0 16 30 16 active+recovery_wait+degraded 2024-09-13T12:12:07.841687+0000 73'46 85:153 [1,0,2] 1 [1,0,2] 1 32'43 2024-09-13T12:09:49.907160+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 no scrub is scheduled 2 0 2024-09-13T12:12:24.131 INFO:tasks.workunit.client.0.smithi032.stdout:1.f 4 0 0 0 0 1932 0 0 14 30 14 active+clean 2024-09-13T12:12:17.209549+0000 38'44 85:149 [1,2,0] 1 [1,2,0] 1 38'44 2024-09-13T12:12:17.209485+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T18:14:52.514325+0000 4 0 2024-09-13T12:12:24.131 INFO:tasks.workunit.client.0.smithi032.stdout:1.10 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:12:07.842621+0000 43'45 85:143 [1,0,2] 1 [1,0,2] 1 34'44 2024-09-13T12:09:52.759350+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:54:31.431312+0000 2 0 2024-09-13T12:12:24.131 INFO:tasks.workunit.client.0.smithi032.stdout:1.11 3 0 1 0 0 1449 0 0 15 30 15 active+recovering+degraded 2024-09-13T12:12:07.844295+0000 34'45 85:153 [2,0,1] 2 [2,0,1] 2 34'45 2024-09-13T12:09:53.421579+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 no scrub is scheduled 3 0 2024-09-13T12:12:24.131 INFO:tasks.workunit.client.0.smithi032.stdout:1.12 1 0 0 0 0 483 0 0 14 30 14 active+clean 2024-09-13T12:12:07.835667+0000 34'44 85:159 [0,1,2] 0 [0,1,2] 0 34'44 2024-09-13T12:09:52.858658+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T17:56:42.489529+0000 1 0 2024-09-13T12:12:24.132 INFO:tasks.workunit.client.0.smithi032.stdout:1.13 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:12:10.075207+0000 44'45 85:166 [1,2,0] 1 [1,2,0] 1 44'45 2024-09-13T12:12:10.075145+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:08:22.156011+0000 3 0 2024-09-13T12:12:24.132 INFO:tasks.workunit.client.0.smithi032.stdout:1.14 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:12:07.845969+0000 44'45 85:145 [1,0,2] 1 [1,0,2] 1 32'43 2024-09-13T12:09:50.727967+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:02:58.594723+0000 1 0 2024-09-13T12:12:24.132 INFO:tasks.workunit.client.0.smithi032.stdout:1.15 3 0 0 0 0 1449 0 0 14 30 14 active+clean 2024-09-13T12:12:07.837370+0000 38'44 85:144 [2,0,1] 2 [2,0,1] 2 38'44 2024-09-13T12:09:54.402988+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T15:25:25.878428+0000 3 0 2024-09-13T12:12:24.132 INFO:tasks.workunit.client.0.smithi032.stdout:1.16 4 0 0 0 0 1932 0 0 16 30 16 active+clean 2024-09-13T12:12:07.843705+0000 38'46 85:147 [1,0,2] 1 [1,0,2] 1 38'46 2024-09-13T12:10:00.648885+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T16:16:10.803318+0000 4 0 2024-09-13T12:12:24.132 INFO:tasks.workunit.client.0.smithi032.stdout:1.17 1 0 0 0 0 483 0 0 13 30 13 active+clean 2024-09-13T12:12:09.031942+0000 32'43 85:141 [1,2,0] 1 [1,2,0] 1 32'43 2024-09-13T12:12:09.031884+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T21:32:21.839825+0000 1 0 2024-09-13T12:12:24.132 INFO:tasks.workunit.client.0.smithi032.stdout:1.18 3 0 0 0 0 1449 0 0 15 30 15 active+clean 2024-09-13T12:12:07.840908+0000 44'45 85:159 [1,2,0] 1 [1,2,0] 1 32'43 2024-09-13T12:09:47.765518+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T23:49:15.964977+0000 1 0 2024-09-13T12:12:24.132 INFO:tasks.workunit.client.0.smithi032.stdout:1.19 2 0 0 0 0 966 0 0 14 30 14 active+clean 2024-09-13T12:12:15.159508+0000 34'44 85:146 [1,2,0] 1 [1,2,0] 1 34'44 2024-09-13T12:12:15.159443+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 periodic scrub scheduled @ 2024-09-14T14:00:50.141731+0000 2 0 2024-09-13T12:12:24.132 INFO:tasks.workunit.client.0.smithi032.stdout:1.1a 4 0 0 0 0 1932 0 0 15 30 15 active+clean 2024-09-13T12:12:07.842209+0000 41'45 85:154 [1,2,0] 1 [1,2,0] 1 41'45 2024-09-13T12:10:05.687638+0000 0'0 2024-09-13T12:09:23.940032+0000 0 1 periodic scrub scheduled @ 2024-09-14T20:27:57.757227+0000 3 0 2024-09-13T12:12:24.133 INFO:tasks.workunit.client.0.smithi032.stdout:1.1b 5 1 1 0 0 2415 0 0 16 30 16 active+recovery_wait+degraded 2024-09-13T12:12:07.843335+0000 44'46 85:176 [1,0,2] 1 [1,0,2] 1 36'45 2024-09-13T12:09:54.677751+0000 0'0 2024-09-13T12:09:23.940032+0000 0 0 no scrub is scheduled 4 0 2024-09-13T12:12:24.133 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:12:24.133 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:12:24.133 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:12:24.141 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: expr 2 + 2 2024-09-13T12:12:24.142 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: seq 1 4 2024-09-13T12:12:24.143 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-13T12:12:24.143 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2024-09-13T12:12:24.179 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-13T12:12:24.179 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2024-09-13T12:12:24.216 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-13T12:12:24.216 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2024-09-13T12:12:24.253 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-13T12:12:24.253 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2024-09-13T12:12:24.290 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: seq 1 2 2024-09-13T12:12:24.290 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-13T12:12:24.290 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2024-09-13T12:12:24.324 INFO:tasks.workunit.client.0.smithi032.stderr:error getting test/newobject_1: (2) No such file or directory 2024-09-13T12:12:24.327 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-13T12:12:24.327 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2024-09-13T12:12:24.361 INFO:tasks.workunit.client.0.smithi032.stderr:error getting test/newobject_2: (2) No such file or directory 2024-09-13T12:12:24.363 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: rm -f td/divergent-priors/existing 2024-09-13T12:12:24.364 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-09-13T12:12:24.365 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-13T12:12:24.385 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.412+0000 7f7472f98640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 51'47 (31'4) delete 1:70b3bf12:::existing_4:head by client.4637.0:1 2024-09-13T12:10:05.314458+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-13T12:12:24.461 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.412+0000 7f7472f98640 20 _merge_object_divergent_entries: keeping 51'47 (31'4) delete 1:70b3bf12:::existing_4:head by client.4637.0:1 2024-09-13T12:10:05.314458+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-13T12:12:24.461 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.412+0000 7f7472f98640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-13T12:12:24.462 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.412+0000 7f7472f98640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 31'4 first_divergent_update: 51'47 last_divergent_update: 51'47 2024-09-13T12:12:24.462 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.412+0000 7f7472f98640 10 _merge_object_divergent_entries: more recent entry found: 73'46 (31'4) modify 1:70b3bf12:::existing_4:head by client.4754.0:1 2024-09-13T12:11:08.015278+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false, already merged 2024-09-13T12:12:24.462 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.412+0000 7f7472f98640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 51'46 (0'0) modify 1:75ac5091:::newobject_2:head by client.4636.0:1 2024-09-13T12:10:05.311294+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-13T12:12:24.462 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.412+0000 7f7472f98640 20 _merge_object_divergent_entries: keeping 51'46 (0'0) modify 1:75ac5091:::newobject_2:head by client.4636.0:1 2024-09-13T12:10:05.311294+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-13T12:12:24.462 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.412+0000 7f7472f98640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-13T12:12:24.462 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.412+0000 7f7472f98640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 51'46 last_divergent_update: 51'46 2024-09-13T12:12:24.462 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.412+0000 7f7472f98640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-09-13T12:12:24.462 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.412+0000 7f7472f98640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-09-13T12:12:24.462 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.545+0000 7f7473f9a640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 51'47 (31'1) modify 1:d858c035:::existing_1:head by client.4639.0:1 2024-09-13T12:10:05.314552+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-13T12:12:24.462 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.545+0000 7f7473f9a640 20 _merge_object_divergent_entries: keeping 51'47 (31'1) modify 1:d858c035:::existing_1:head by client.4639.0:1 2024-09-13T12:10:05.314552+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-13T12:12:24.462 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.545+0000 7f7473f9a640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-13T12:12:24.462 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.545+0000 7f7473f9a640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 31'1 first_divergent_update: 51'47 last_divergent_update: 51'47 2024-09-13T12:12:24.463 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.545+0000 7f7473f9a640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-09-13T12:12:24.463 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.545+0000 7f7473f9a640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-09-13T12:12:24.463 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.545+0000 7f7473f9a640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 51'48 2024-09-13T12:12:24.463 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.545+0000 7f7473f9a640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 51'47 (31'1) modify 1:d858c035:::existing_1:head by client.4639.0:1 2024-09-13T12:10:05.314552+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-13T12:12:24.463 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.545+0000 7f7473f9a640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-09-13T12:12:24.463 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.545+0000 7f7473f9a640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 31'1 <= info.log_tail 31'30 2024-09-13T12:12:24.463 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.545+0000 7f7473f9a640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 51'48 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4638.0:1 2024-09-13T12:10:05.315034+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-13T12:12:24.463 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.545+0000 7f7473f9a640 20 _merge_object_divergent_entries: keeping 51'48 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4638.0:1 2024-09-13T12:10:05.315034+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-13T12:12:24.463 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.545+0000 7f7473f9a640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-13T12:12:24.463 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.545+0000 7f7473f9a640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 51'48 last_divergent_update: 51'48 2024-09-13T12:12:24.463 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.545+0000 7f7473f9a640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-09-13T12:12:24.463 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.545+0000 7f7473f9a640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-09-13T12:12:24.463 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.546+0000 7f7472f98640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 51'46 (31'2) modify 1:97db2a0f:::existing_2:head by client.4635.0:1 2024-09-13T12:10:05.313002+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-13T12:12:24.464 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.546+0000 7f7472f98640 20 _merge_object_divergent_entries: keeping 51'46 (31'2) modify 1:97db2a0f:::existing_2:head by client.4635.0:1 2024-09-13T12:10:05.313002+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-13T12:12:24.464 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.546+0000 7f7472f98640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-13T12:12:24.464 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.546+0000 7f7472f98640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 31'2 first_divergent_update: 51'46 last_divergent_update: 51'46 2024-09-13T12:12:24.464 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.546+0000 7f7472f98640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-09-13T12:12:24.464 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.546+0000 7f7472f98640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-09-13T12:12:24.464 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.546+0000 7f7472f98640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 51'46 2024-09-13T12:12:24.465 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.546+0000 7f7472f98640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 51'46 (31'2) modify 1:97db2a0f:::existing_2:head by client.4635.0:1 2024-09-13T12:10:05.313002+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-13T12:12:24.468 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.546+0000 7f7472f98640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-09-13T12:12:24.468 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:54.546+0000 7f7472f98640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 31'2 <= info.log_tail 31'30 2024-09-13T12:12:24.468 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:55.333+0000 7f7473f9a640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 51'46 (31'3) delete 1:8881e6d8:::existing_3:head by client.4634.0:1 2024-09-13T12:10:05.313950+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-13T12:12:24.468 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:55.333+0000 7f7473f9a640 20 _merge_object_divergent_entries: keeping 51'46 (31'3) delete 1:8881e6d8:::existing_3:head by client.4634.0:1 2024-09-13T12:10:05.313950+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-13T12:12:24.468 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:55.333+0000 7f7473f9a640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-09-13T12:12:24.468 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:55.333+0000 7f7473f9a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 31'3 first_divergent_update: 51'46 last_divergent_update: 51'46 2024-09-13T12:12:24.469 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:55.333+0000 7f7473f9a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-09-13T12:12:24.469 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:55.333+0000 7f7473f9a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-09-13T12:12:24.469 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:55.333+0000 7f7473f9a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 51'46 2024-09-13T12:12:24.469 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:55.333+0000 7f7473f9a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 51'46 (31'3) delete 1:8881e6d8:::existing_3:head by client.4634.0:1 2024-09-13T12:10:05.313950+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-13T12:12:24.469 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:55.333+0000 7f7473f9a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-09-13T12:12:24.469 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:11:55.333+0000 7f7473f9a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 31'3 <= info.log_tail 31'30 2024-09-13T12:12:24.469 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.2.log:2024-09-13T12:11:54.528+0000 7f203e907640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 51'46 (31'3) delete 1:8881e6d8:::existing_3:head by client.4634.0:1 2024-09-13T12:10:05.313950+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-13T12:12:24.469 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.2.log:2024-09-13T12:11:54.528+0000 7f203e907640 20 _merge_object_divergent_entries: keeping 51'46 (31'3) delete 1:8881e6d8:::existing_3:head by client.4634.0:1 2024-09-13T12:10:05.313950+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-13T12:12:24.469 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.2.log:2024-09-13T12:11:54.528+0000 7f203e907640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-09-13T12:12:24.469 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.2.log:2024-09-13T12:11:54.528+0000 7f203e907640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 31'3 first_divergent_update: 51'46 last_divergent_update: 51'46 2024-09-13T12:12:24.469 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.2.log:2024-09-13T12:11:54.528+0000 7f203e907640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-09-13T12:12:24.469 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.2.log:2024-09-13T12:11:54.528+0000 7f203e907640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-09-13T12:12:24.469 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.2.log:2024-09-13T12:11:54.528+0000 7f203e907640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 51'46 2024-09-13T12:12:24.470 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.2.log:2024-09-13T12:11:54.528+0000 7f203e907640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 51'46 (31'3) delete 1:8881e6d8:::existing_3:head by client.4634.0:1 2024-09-13T12:10:05.313950+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-13T12:12:24.470 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.2.log:2024-09-13T12:11:54.528+0000 7f203e907640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-09-13T12:12:24.470 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.2.log:2024-09-13T12:11:54.528+0000 7f203e907640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 31'3 <= info.log_tail 31'30 2024-09-13T12:12:24.470 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-09-13T12:12:24.470 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-13T12:12:24.470 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-09-13T12:12:24.471 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-13T12:12:24.471 INFO:tasks.workunit.client.0.smithi032.stdout:success 2024-09-13T12:12:24.473 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:842: TEST_divergent_3: echo success 2024-09-13T12:12:24.473 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:844: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.33242.out 2024-09-13T12:12:24.473 INFO:tasks.workunit.client.0.smithi032.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.33242.out': No such file or directory 2024-09-13T12:12:24.473 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:846: TEST_divergent_3: delete_pool test 2024-09-13T12:12:24.473 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test 2024-09-13T12:12:24.473 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-13T12:12:24.939 INFO:tasks.workunit.client.0.smithi032.stderr:pool 'test' does not exist 2024-09-13T12:12:24.951 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:847: TEST_divergent_3: kill_daemons td/divergent-priors 2024-09-13T12:12:24.952 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:12:24.952 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:12:24.952 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:12:24.952 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:12:24.952 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:12:30.265 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:12:30.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-13T12:12:30.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-13T12:12:30.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T12:12:30.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-13T12:12:30.266 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:12:30.266 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:12:30.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:12:30.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:12:30.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:12:30.272 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:12:30.273 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T12:12:30.274 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T12:12:30.274 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T12:12:30.275 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T12:12:30.275 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T12:12:30.276 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T12:12:30.277 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:12:30.277 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T12:12:30.278 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T12:12:30.278 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:12:30.278 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:12:30.279 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T12:12:30.280 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-13T12:12:30.317 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T12:12:30.317 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:12:30.317 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:12:30.317 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.33242 2024-09-13T12:12:30.319 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T12:12:30.319 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T12:12:30.319 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-13T12:12:30.319 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-13T12:12:30.319 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/divergent-priors 2024-09-13T12:12:30.319 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/divergent-priors 2024-09-13T12:12:30.319 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-13T12:12:30.319 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T12:12:30.319 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-13T12:12:30.319 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:12:30.319 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:12:30.320 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:12:30.320 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:12:30.320 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:12:30.320 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:12:30.321 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T12:12:30.322 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T12:12:30.322 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T12:12:30.323 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T12:12:30.323 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T12:12:30.324 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T12:12:30.324 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:12:30.324 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T12:12:30.325 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T12:12:30.325 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:12:30.326 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:12:30.327 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T12:12:30.327 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-13T12:12:30.329 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T12:12:30.329 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:12:30.329 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:12:30.329 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.33242 2024-09-13T12:12:30.329 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T12:12:30.330 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T12:12:30.330 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/divergent-priors 2024-09-13T12:12:30.331 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T12:12:30.331 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:12:30.331 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:12:30.331 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.33242 2024-09-13T12:12:30.333 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T12:12:30.333 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T12:12:30.333 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T12:12:30.333 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-13T12:12:30.334 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2024-09-13T12:12:30.334 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2024-09-13T12:12:30.334 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2024-09-13T12:12:30.334 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2024-09-13T12:12:30.334 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2024-09-13T12:12:30.334 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2024-09-13T12:12:30.334 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2024-09-13T12:12:30.336 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2024-09-13T12:12:30.336 INFO:tasks.workunit.client.0.smithi032.stderr:1 2024-09-13T12:12:30.336 INFO:tasks.workunit.client.0.smithi032.stderr:2' 2024-09-13T12:12:30.336 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2024-09-13T12:12:30.336 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/divergent-priors 2024-09-13T12:12:30.336 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T12:12:30.336 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T12:12:30.336 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T12:12:30.337 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/divergent-priors/a 2024-09-13T12:12:30.337 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-13T12:12:30.805 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T12:12:30.806 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:12:30.806 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:12:30.806 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:12:30.806 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:12:30.806 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:12:30.806 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:12:30.806 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-13T12:12:31.015 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T12:12:31.016 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T12:12:31.016 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T12:12:31.016 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T12:12:31.016 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T12:12:31.017 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T12:12:31.017 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T12:12:31.017 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T12:12:31.017 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T12:12:31.017 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:12:31.017 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:12:31.017 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:12:31.017 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-mon.a.asok 2024-09-13T12:12:31.018 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T12:12:31.018 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.33242/ceph-mon.a.asok config get fsid 2024-09-13T12:12:31.109 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T12:12:31.109 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T12:12:31.109 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T12:12:31.109 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T12:12:31.109 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T12:12:31.109 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T12:12:31.109 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T12:12:31.109 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T12:12:31.110 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:12:31.110 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:12:31.110 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:12:31.110 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-mon.a.asok 2024-09-13T12:12:31.110 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T12:12:31.110 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.33242/ceph-mon.a.asok config get mon_host 2024-09-13T12:12:31.192 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2024-09-13T12:12:31.192 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/divergent-priors 2024-09-13T12:12:31.193 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T12:12:31.193 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T12:12:31.193 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T12:12:31.193 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/divergent-priors/x 2024-09-13T12:12:31.193 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T12:12:32.056 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T12:12:32.057 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:12:32.057 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:12:32.057 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:12:32.057 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:12:32.057 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:12:32.057 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:12:32.058 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T12:12:32.058 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T12:12:32.095 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-09-13T12:12:32.095 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2024-09-13T12:12:32.095 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-13T12:12:32.095 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T12:12:32.095 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T12:12:32.095 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T12:12:32.095 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/0 2024-09-13T12:12:32.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T12:12:32.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:12:32.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:12:32.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:12:32.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-13T12:12:32.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-13T12:12:32.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T12:12:32.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T12:12:32.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T12:12:32.096 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T12:12:32.097 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:12:32.097 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:12:32.097 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:12:32.097 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:12:32.097 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:12:32.097 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:12:32.097 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:12:32.097 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:12:32.097 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T12:12:32.097 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T12:12:32.098 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T12:12:32.098 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T12:12:32.098 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:12:32.098 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:12:32.098 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:12:32.098 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:12:32.098 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T12:12:32.099 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T12:12:32.099 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/0 2024-09-13T12:12:32.099 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T12:12:32.100 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=92bce19c-5a5e-4360-a088-1bf366fc2ecf 2024-09-13T12:12:32.100 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 92bce19c-5a5e-4360-a088-1bf366fc2ecf' 2024-09-13T12:12:32.101 INFO:tasks.workunit.client.0.smithi032.stdout:add osd0 92bce19c-5a5e-4360-a088-1bf366fc2ecf 2024-09-13T12:12:32.101 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T12:12:32.122 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAwLORmQ2giBxAAoYlgN+Ktx4QomubAvOJQ7A== 2024-09-13T12:12:32.122 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAwLORmQ2giBxAAoYlgN+Ktx4QomubAvOJQ7A=="}' 2024-09-13T12:12:32.122 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 92bce19c-5a5e-4360-a088-1bf366fc2ecf -i td/divergent-priors/0/new.json 2024-09-13T12:12:32.396 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:12:32.406 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/0/new.json 2024-09-13T12:12:32.407 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAwLORmQ2giBxAAoYlgN+Ktx4QomubAvOJQ7A== --osd-uuid 92bce19c-5a5e-4360-a088-1bf366fc2ecf 2024-09-13T12:12:32.445 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:12:32.444+0000 7f6f35a2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:12:32.465 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:12:32.464+0000 7f6f35a2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:12:32.466 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:12:32.465+0000 7f6f35a2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:12:32.466 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:12:32.466+0000 7f6f35a2c640 -1 bdev(0x560137e07c00 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-09-13T12:12:32.466 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:12:32.466+0000 7f6f35a2c640 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-13T12:12:38.069 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-13T12:12:38.069 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T12:12:38.070 INFO:tasks.workunit.client.0.smithi032.stdout:adding osd0 key to auth repository 2024-09-13T12:12:38.071 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T12:12:38.071 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T12:12:38.641 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.0 2024-09-13T12:12:38.642 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T12:12:38.642 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:12:38.642 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T12:12:38.642 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T12:12:38.642 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T12:12:38.677 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:12:38.676+0000 7fba13c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:12:38.679 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:12:38.678+0000 7fba13c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:12:38.680 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:12:38.679+0000 7fba13c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:12:39.070 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T12:12:39.071 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:12:39.071 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T12:12:39.071 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:12:39.071 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:12:39.071 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:12:39.071 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:12:39.072 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:12:39.072 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:12:39.072 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:12:39.516 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:12:40.017 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:12:40.016+0000 7fba13c0d640 -1 Falling back to public interface 2024-09-13T12:12:40.517 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:12:40.518 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:12:40.518 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:12:40.518 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:12:40.518 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:12:40.519 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:12:40.967 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:12:41.969 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:12:41.969 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:12:41.969 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:12:41.970 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:12:41.970 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:12:41.970 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:12:42.388 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:12:42.503 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:12:42.502+0000 7fba13c0d640 -1 osd.0 0 log_to_monitors true 2024-09-13T12:12:43.389 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:12:43.389 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:12:43.389 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:12:43.390 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:12:43.390 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:12:43.390 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:12:43.866 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:12:44.867 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:12:44.867 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:12:44.867 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:12:44.868 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:12:44.868 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:12:44.868 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:12:45.332 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:12:46.333 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:12:46.333 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:12:46.334 INFO:tasks.workunit.client.0.smithi032.stdout:5 2024-09-13T12:12:46.334 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T12:12:46.334 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:12:46.334 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:12:46.780 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:12:47.781 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:12:47.781 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:12:47.781 INFO:tasks.workunit.client.0.smithi032.stdout:6 2024-09-13T12:12:47.782 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T12:12:47.782 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:12:47.782 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:12:48.342 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:12:49.343 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:12:49.343 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:12:49.344 INFO:tasks.workunit.client.0.smithi032.stdout:7 2024-09-13T12:12:49.344 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T12:12:49.344 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:12:49.345 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:12:49.771 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:12:50.772 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:12:50.773 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:12:50.773 INFO:tasks.workunit.client.0.smithi032.stdout:8 2024-09-13T12:12:50.774 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T12:12:50.774 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:12:50.774 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:12:51.316 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:12:52.318 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:12:52.318 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:12:52.318 INFO:tasks.workunit.client.0.smithi032.stdout:9 2024-09-13T12:12:52.318 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T12:12:52.319 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:12:52.319 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:12:52.741 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:12:53.742 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:12:53.742 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:12:53.743 INFO:tasks.workunit.client.0.smithi032.stdout:10 2024-09-13T12:12:53.743 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T12:12:53.743 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:12:53.743 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:12:54.279 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:12:55.280 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:12:55.281 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:12:55.281 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T12:12:55.281 INFO:tasks.workunit.client.0.smithi032.stdout:11 2024-09-13T12:12:55.281 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:12:55.281 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:12:56.086 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:12:57.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:12:57.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:12:57.088 INFO:tasks.workunit.client.0.smithi032.stdout:12 2024-09-13T12:12:57.088 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T12:12:57.088 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:12:57.088 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:12:57.806 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:12:58.808 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:12:58.808 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:12:58.808 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T12:12:58.809 INFO:tasks.workunit.client.0.smithi032.stdout:13 2024-09-13T12:12:58.809 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:12:58.809 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:12:59.607 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:13:00.608 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:13:00.608 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:00.608 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T12:13:00.609 INFO:tasks.workunit.client.0.smithi032.stdout:14 2024-09-13T12:13:00.609 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:00.609 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:13:01.381 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:13:02.382 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:13:02.382 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:02.383 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T12:13:02.383 INFO:tasks.workunit.client.0.smithi032.stdout:15 2024-09-13T12:13:02.383 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:02.383 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:13:03.432 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:13:04.433 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:13:04.434 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:04.434 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T12:13:04.434 INFO:tasks.workunit.client.0.smithi032.stdout:16 2024-09-13T12:13:04.434 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:04.434 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:13:05.188 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:13:06.190 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:13:06.190 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:06.190 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T12:13:06.190 INFO:tasks.workunit.client.0.smithi032.stdout:17 2024-09-13T12:13:06.191 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:06.191 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:13:06.643 INFO:tasks.workunit.client.0.smithi032.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3373456545,v1:127.0.0.1:6803/3373456545] [v2:127.0.0.1:6804/3373456545,v1:127.0.0.1:6805/3373456545] exists,up 92bce19c-5a5e-4360-a088-1bf366fc2ecf 2024-09-13T12:13:06.647 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:13:06.647 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:13:06.647 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:13:06.647 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-09-13T12:13:06.647 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2024-09-13T12:13:06.647 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-13T12:13:06.647 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T12:13:06.647 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T12:13:06.647 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T12:13:06.648 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/1 2024-09-13T12:13:06.648 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T12:13:06.648 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:13:06.648 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:13:06.648 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:13:06.648 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-13T12:13:06.648 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-13T12:13:06.648 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T12:13:06.648 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T12:13:06.648 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T12:13:06.648 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T12:13:06.648 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:13:06.648 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:13:06.649 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:13:06.649 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:13:06.649 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:13:06.649 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:13:06.649 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:13:06.649 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:13:06.649 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T12:13:06.649 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T12:13:06.649 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T12:13:06.649 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T12:13:06.649 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:13:06.649 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:13:06.649 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:13:06.650 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:13:06.650 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T12:13:06.650 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T12:13:06.650 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/1 2024-09-13T12:13:06.650 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T12:13:06.650 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b6fb1428-41f7-46eb-8099-15ef8f53355b 2024-09-13T12:13:06.650 INFO:tasks.workunit.client.0.smithi032.stdout:add osd1 b6fb1428-41f7-46eb-8099-15ef8f53355b 2024-09-13T12:13:06.650 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 b6fb1428-41f7-46eb-8099-15ef8f53355b' 2024-09-13T12:13:06.651 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T12:13:06.672 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBSLORmg6n7JxAA4eRnsj2LYgeI+QM67DIVEA== 2024-09-13T12:13:06.672 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBSLORmg6n7JxAA4eRnsj2LYgeI+QM67DIVEA=="}' 2024-09-13T12:13:06.672 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b6fb1428-41f7-46eb-8099-15ef8f53355b -i td/divergent-priors/1/new.json 2024-09-13T12:13:07.224 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:13:07.236 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/1/new.json 2024-09-13T12:13:07.237 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBSLORmg6n7JxAA4eRnsj2LYgeI+QM67DIVEA== --osd-uuid b6fb1428-41f7-46eb-8099-15ef8f53355b 2024-09-13T12:13:07.274 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:13:07.273+0000 7f8c885d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:13:07.276 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:13:07.275+0000 7f8c885d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:13:07.277 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:13:07.276+0000 7f8c885d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:13:07.278 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:13:07.277+0000 7f8c885d9640 -1 bdev(0x55dc85eafc00 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-09-13T12:13:07.278 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:13:07.277+0000 7f8c885d9640 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-13T12:13:13.001 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-13T12:13:13.001 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T12:13:13.001 INFO:tasks.workunit.client.0.smithi032.stdout:adding osd1 key to auth repository 2024-09-13T12:13:13.002 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T12:13:13.002 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T12:13:13.569 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.1 2024-09-13T12:13:13.570 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T12:13:13.570 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:13:13.570 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T12:13:13.570 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T12:13:13.571 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T12:13:13.606 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:13:13.605+0000 7fbecbc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:13:13.608 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:13:13.607+0000 7fbecbc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:13:13.609 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:13:13.608+0000 7fbecbc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:13:14.017 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T12:13:14.017 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:13:14.017 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T12:13:14.017 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:13:14.017 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:13:14.017 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:14.017 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:13:14.018 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:13:14.018 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:14.018 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:13:14.469 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:13:14.965 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:13:14.964+0000 7fbecbc0d640 -1 Falling back to public interface 2024-09-13T12:13:15.470 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:13:15.470 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:15.470 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:13:15.471 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:13:15.471 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:15.471 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:13:15.918 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:13:16.920 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:13:16.920 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:16.920 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:13:16.921 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:13:16.921 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:16.921 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:13:17.370 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:13:17.833 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:13:17.832+0000 7fbecbc0d640 -1 osd.1 0 log_to_monitors true 2024-09-13T12:13:18.371 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:13:18.372 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:18.372 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:13:18.372 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:13:18.372 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:18.373 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:13:18.970 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:13:19.972 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:13:19.972 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:19.972 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:13:19.972 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:13:19.973 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:19.973 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:13:20.482 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:13:21.484 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:13:21.484 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:21.484 INFO:tasks.workunit.client.0.smithi032.stdout:5 2024-09-13T12:13:21.485 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T12:13:21.485 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:21.485 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:13:21.936 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:13:22.936 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:13:22.936 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:22.936 INFO:tasks.workunit.client.0.smithi032.stdout:6 2024-09-13T12:13:22.937 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T12:13:22.937 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:22.937 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:13:23.491 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:13:24.493 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:13:24.493 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:24.493 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T12:13:24.493 INFO:tasks.workunit.client.0.smithi032.stdout:7 2024-09-13T12:13:24.494 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:24.494 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:13:25.001 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:13:26.004 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:13:26.005 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:26.005 INFO:tasks.workunit.client.0.smithi032.stdout:8 2024-09-13T12:13:26.006 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T12:13:26.006 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:26.006 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:13:26.556 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:13:27.557 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:13:27.557 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:27.557 INFO:tasks.workunit.client.0.smithi032.stdout:9 2024-09-13T12:13:27.558 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T12:13:27.558 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:27.558 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:13:28.061 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:13:29.062 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:13:29.063 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:29.063 INFO:tasks.workunit.client.0.smithi032.stdout:10 2024-09-13T12:13:29.063 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T12:13:29.063 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:29.064 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:13:29.569 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:13:30.571 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:13:30.571 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:30.571 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T12:13:30.571 INFO:tasks.workunit.client.0.smithi032.stdout:11 2024-09-13T12:13:30.572 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:30.572 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:13:31.231 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:13:32.232 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:13:32.232 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:32.233 INFO:tasks.workunit.client.0.smithi032.stdout:12 2024-09-13T12:13:32.233 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T12:13:32.233 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:32.233 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:13:32.798 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:13:33.799 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:13:33.799 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:33.799 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T12:13:33.799 INFO:tasks.workunit.client.0.smithi032.stdout:13 2024-09-13T12:13:33.800 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:33.800 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:13:34.556 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:13:35.558 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:13:35.558 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:35.558 INFO:tasks.workunit.client.0.smithi032.stdout:14 2024-09-13T12:13:35.559 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T12:13:35.559 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:35.559 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:13:36.313 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:13:37.314 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:13:37.314 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:37.314 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T12:13:37.314 INFO:tasks.workunit.client.0.smithi032.stdout:15 2024-09-13T12:13:37.315 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:37.315 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:13:38.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:13:39.159 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:13:39.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:39.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T12:13:39.160 INFO:tasks.workunit.client.0.smithi032.stdout:16 2024-09-13T12:13:39.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:39.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:13:39.891 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:13:40.892 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:13:40.892 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:40.892 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T12:13:40.893 INFO:tasks.workunit.client.0.smithi032.stdout:17 2024-09-13T12:13:40.893 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:40.893 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:13:41.820 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:13:42.821 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:13:42.822 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:42.822 INFO:tasks.workunit.client.0.smithi032.stdout:18 2024-09-13T12:13:42.822 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T12:13:42.823 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:42.823 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:13:43.242 INFO:tasks.workunit.client.0.smithi032.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3755373257,v1:127.0.0.1:6811/3755373257] [v2:127.0.0.1:6812/3755373257,v1:127.0.0.1:6813/3755373257] exists,up b6fb1428-41f7-46eb-8099-15ef8f53355b 2024-09-13T12:13:43.246 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:13:43.246 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:13:43.246 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:13:43.246 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-09-13T12:13:43.246 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2024-09-13T12:13:43.246 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-13T12:13:43.246 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T12:13:43.246 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T12:13:43.246 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T12:13:43.246 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/2 2024-09-13T12:13:43.246 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T12:13:43.247 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:13:43.247 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:13:43.247 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:13:43.247 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-13T12:13:43.247 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-13T12:13:43.247 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T12:13:43.247 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T12:13:43.248 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T12:13:43.248 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T12:13:43.248 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:13:43.248 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:13:43.248 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:13:43.248 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:13:43.248 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:13:43.248 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:13:43.249 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:13:43.249 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:13:43.249 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T12:13:43.249 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T12:13:43.249 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T12:13:43.249 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T12:13:43.249 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:13:43.249 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:13:43.249 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:13:43.249 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:13:43.249 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T12:13:43.249 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T12:13:43.249 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/2 2024-09-13T12:13:43.250 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T12:13:43.250 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=575c91a2-dd41-4141-bbec-f3711b8a1a2e 2024-09-13T12:13:43.250 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 575c91a2-dd41-4141-bbec-f3711b8a1a2e' 2024-09-13T12:13:43.250 INFO:tasks.workunit.client.0.smithi032.stdout:add osd2 575c91a2-dd41-4141-bbec-f3711b8a1a2e 2024-09-13T12:13:43.250 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T12:13:43.270 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB3LORmfN4REBAA7p3e8uVuaIy1AZrbfk4coQ== 2024-09-13T12:13:43.270 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB3LORmfN4REBAA7p3e8uVuaIy1AZrbfk4coQ=="}' 2024-09-13T12:13:43.270 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 575c91a2-dd41-4141-bbec-f3711b8a1a2e -i td/divergent-priors/2/new.json 2024-09-13T12:13:43.763 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:13:43.774 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/2/new.json 2024-09-13T12:13:43.775 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB3LORmfN4REBAA7p3e8uVuaIy1AZrbfk4coQ== --osd-uuid 575c91a2-dd41-4141-bbec-f3711b8a1a2e 2024-09-13T12:13:43.813 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:13:43.811+0000 7f1c20bcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:13:43.826 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:13:43.824+0000 7f1c20bcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:13:43.826 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:13:43.825+0000 7f1c20bcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:13:43.827 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:13:43.825+0000 7f1c20bcc640 -1 bdev(0x55b84bcafc00 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-09-13T12:13:43.827 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:13:43.825+0000 7f1c20bcc640 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-13T12:13:49.847 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-13T12:13:49.848 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T12:13:49.848 INFO:tasks.workunit.client.0.smithi032.stdout:adding osd2 key to auth repository 2024-09-13T12:13:49.849 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T12:13:49.849 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T12:13:50.431 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.2 2024-09-13T12:13:50.432 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T12:13:50.432 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:13:50.432 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T12:13:50.432 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T12:13:50.433 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T12:13:50.467 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:13:50.465+0000 7fc3b6464640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:13:50.488 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:13:50.486+0000 7fc3b6464640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:13:50.489 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:13:50.487+0000 7fc3b6464640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:13:50.910 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T12:13:50.910 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:13:50.910 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T12:13:50.911 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:13:50.911 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:13:50.911 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:50.911 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:13:50.911 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:13:50.912 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:50.912 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:13:51.342 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:13:51.844 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:13:51.842+0000 7fc3b6464640 -1 Falling back to public interface 2024-09-13T12:13:52.344 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:13:52.344 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:52.344 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:13:52.345 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:13:52.345 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:52.345 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:13:52.796 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:13:53.797 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:13:53.798 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:53.798 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:13:53.798 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:13:53.798 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:53.799 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:13:54.216 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:13:54.777 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:13:54.776+0000 7fc3b6464640 -1 osd.2 0 log_to_monitors true 2024-09-13T12:13:55.217 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:13:55.217 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:55.218 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:13:55.218 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:13:55.218 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:55.218 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:13:55.662 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:13:56.663 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:13:56.663 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:56.663 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:13:56.664 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:13:56.664 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:56.664 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:13:57.101 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:13:58.103 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:13:58.103 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:58.103 INFO:tasks.workunit.client.0.smithi032.stdout:5 2024-09-13T12:13:58.104 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T12:13:58.104 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:58.104 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:13:58.651 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:13:59.652 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:13:59.652 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:13:59.652 INFO:tasks.workunit.client.0.smithi032.stdout:6 2024-09-13T12:13:59.653 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T12:13:59.653 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:13:59.653 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:14:00.080 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:14:01.082 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:14:01.082 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:14:01.082 INFO:tasks.workunit.client.0.smithi032.stdout:7 2024-09-13T12:14:01.082 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T12:14:01.083 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:14:01.083 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:14:01.548 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:14:02.550 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:14:02.550 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:14:02.550 INFO:tasks.workunit.client.0.smithi032.stdout:8 2024-09-13T12:14:02.550 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T12:14:02.551 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:14:02.551 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:14:03.101 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:14:04.102 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:14:04.102 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:14:04.103 INFO:tasks.workunit.client.0.smithi032.stdout:9 2024-09-13T12:14:04.103 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T12:14:04.103 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:14:04.103 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:14:04.526 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:14:05.527 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:14:05.527 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:14:05.527 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T12:14:05.527 INFO:tasks.workunit.client.0.smithi032.stdout:10 2024-09-13T12:14:05.528 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:14:05.528 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:14:06.066 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:14:07.067 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:14:07.067 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:14:07.068 INFO:tasks.workunit.client.0.smithi032.stdout:11 2024-09-13T12:14:07.068 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T12:14:07.068 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:14:07.068 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:14:08.019 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:14:09.021 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:14:09.021 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:14:09.021 INFO:tasks.workunit.client.0.smithi032.stdout:12 2024-09-13T12:14:09.022 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T12:14:09.022 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:14:09.022 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:14:10.208 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:14:11.210 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:14:11.210 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:14:11.210 INFO:tasks.workunit.client.0.smithi032.stdout:13 2024-09-13T12:14:11.211 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T12:14:11.211 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:14:11.211 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:14:11.972 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:14:12.974 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:14:12.974 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:14:12.974 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T12:14:12.974 INFO:tasks.workunit.client.0.smithi032.stdout:14 2024-09-13T12:14:12.975 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:14:12.975 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:14:13.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:14:14.722 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:14:14.722 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:14:14.722 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T12:14:14.722 INFO:tasks.workunit.client.0.smithi032.stdout:15 2024-09-13T12:14:14.723 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:14:14.723 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:14:15.751 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:14:16.752 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:14:16.752 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:14:16.753 INFO:tasks.workunit.client.0.smithi032.stdout:16 2024-09-13T12:14:16.753 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T12:14:16.753 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:14:16.753 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:14:17.586 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:14:18.588 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:14:18.588 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:14:18.588 INFO:tasks.workunit.client.0.smithi032.stdout:17 2024-09-13T12:14:18.589 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T12:14:18.589 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:14:18.589 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:14:19.055 INFO:tasks.workunit.client.0.smithi032.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/591555181,v1:127.0.0.1:6819/591555181] [v2:127.0.0.1:6820/591555181,v1:127.0.0.1:6821/591555181] exists,up 575c91a2-dd41-4141-bbec-f3711b8a1a2e 2024-09-13T12:14:19.056 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:14:19.056 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:14:19.056 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:14:19.056 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2024-09-13T12:14:19.679 INFO:tasks.workunit.client.0.smithi032.stderr:noout is set 2024-09-13T12:14:19.691 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2024-09-13T12:14:20.483 INFO:tasks.workunit.client.0.smithi032.stderr:noin is set 2024-09-13T12:14:20.495 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2024-09-13T12:14:21.521 INFO:tasks.workunit.client.0.smithi032.stderr:nodown is set 2024-09-13T12:14:21.533 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2024-09-13T12:14:21.533 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=test 2024-09-13T12:14:21.533 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-13T12:14:21.534 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=true 2024-09-13T12:14:21.534 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-13T12:14:21.534 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-13T12:14:22.128 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool test 1 1 erasure myprofile 2024-09-13T12:14:22.128 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 erasure myprofile 2024-09-13T12:14:23.069 INFO:tasks.workunit.client.0.smithi032.stderr:pool 'test' already exists 2024-09-13T12:14:23.080 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T12:14:24.082 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' true = true ']' 2024-09-13T12:14:24.082 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2024-09-13T12:14:24.825 INFO:tasks.workunit.client.0.smithi032.stderr:set pool 1 allow_ec_overwrites to true 2024-09-13T12:14:24.838 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-13T12:14:24.838 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T12:14:24.839 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T12:14:24.839 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T12:14:24.839 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T12:14:24.839 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T12:14:24.839 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T12:14:24.839 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T12:14:24.839 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T12:14:24.839 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T12:14:24.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T12:14:24.932 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T12:14:24.932 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T12:14:24.932 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T12:14:24.932 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T12:14:24.932 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T12:14:25.340 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T12:14:25.340 INFO:tasks.workunit.client.0.smithi032.stderr:1 2024-09-13T12:14:25.340 INFO:tasks.workunit.client.0.smithi032.stderr:2' 2024-09-13T12:14:25.340 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T12:14:25.340 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:14:25.340 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T12:14:25.465 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836504 2024-09-13T12:14:25.465 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836504 2024-09-13T12:14:25.465 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504' 2024-09-13T12:14:25.465 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:14:25.465 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T12:14:25.590 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672973 2024-09-13T12:14:25.590 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672973 2024-09-13T12:14:25.590 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672973' 2024-09-13T12:14:25.590 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:14:25.590 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T12:14:25.715 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509446 2024-09-13T12:14:25.715 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509446 2024-09-13T12:14:25.715 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672973 2-64424509446' 2024-09-13T12:14:25.715 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:14:25.715 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836504 2024-09-13T12:14:25.716 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:14:25.716 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T12:14:25.717 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836504 2024-09-13T12:14:25.717 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:14:25.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836504 2024-09-13T12:14:25.718 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.0 seq 21474836504 2024-09-13T12:14:25.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-13T12:14:25.718 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:14:26.151 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836504 2024-09-13T12:14:26.151 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T12:14:27.152 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T12:14:27.152 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:14:27.612 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836505 -lt 21474836504 2024-09-13T12:14:27.612 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:14:27.613 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672973 2024-09-13T12:14:27.613 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:14:27.614 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T12:14:27.614 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672973 2024-09-13T12:14:27.614 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:14:27.615 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672973 2024-09-13T12:14:27.615 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-09-13T12:14:27.616 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.1 seq 42949672973 2024-09-13T12:14:27.616 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T12:14:28.071 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672973 -lt 42949672973 2024-09-13T12:14:28.072 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:14:28.072 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509446 2024-09-13T12:14:28.072 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:14:28.073 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T12:14:28.074 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509446 2024-09-13T12:14:28.074 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:14:28.074 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509446 2024-09-13T12:14:28.075 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.2 seq 64424509446 2024-09-13T12:14:28.075 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-09-13T12:14:28.075 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T12:14:28.514 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509446 -lt 64424509446 2024-09-13T12:14:28.514 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T12:14:28.514 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:14:28.515 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:14:29.085 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T12:14:29.086 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:14:29.086 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:14:29.086 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:14:29.086 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:14:29.086 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:14:29.086 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:14:29.086 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:14:29.490 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T12:14:29.491 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:14:29.491 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:14:29.491 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:14:30.042 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T12:14:30.042 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T12:14:30.042 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T12:14:30.042 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-13T12:14:30.042 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2024-09-13T12:14:30.042 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T12:14:30.042 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T12:14:30.464 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T12:14:30.464 INFO:tasks.workunit.client.0.smithi032.stderr:1 2024-09-13T12:14:30.464 INFO:tasks.workunit.client.0.smithi032.stderr:2' 2024-09-13T12:14:30.464 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T12:14:30.464 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:14:30.464 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T12:14:30.619 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836507 2024-09-13T12:14:30.619 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836507 2024-09-13T12:14:30.619 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507' 2024-09-13T12:14:30.619 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:14:30.619 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T12:14:30.744 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672976 2024-09-13T12:14:30.744 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672976 2024-09-13T12:14:30.744 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-42949672976' 2024-09-13T12:14:30.744 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:14:30.744 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T12:14:30.869 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509449 2024-09-13T12:14:30.869 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509449 2024-09-13T12:14:30.869 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-42949672976 2-64424509449' 2024-09-13T12:14:30.869 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:14:30.869 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836507 2024-09-13T12:14:30.870 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:14:30.870 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T12:14:30.871 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836507 2024-09-13T12:14:30.871 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:14:30.872 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836507 2024-09-13T12:14:30.872 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.0 seq 21474836507 2024-09-13T12:14:30.872 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-09-13T12:14:30.872 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:14:31.281 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836505 -lt 21474836507 2024-09-13T12:14:31.281 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T12:14:32.283 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T12:14:32.283 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:14:32.746 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836508 -lt 21474836507 2024-09-13T12:14:32.746 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:14:32.747 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672976 2024-09-13T12:14:32.747 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:14:32.748 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T12:14:32.748 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672976 2024-09-13T12:14:32.748 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:14:32.749 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672976 2024-09-13T12:14:32.749 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.1 seq 42949672976 2024-09-13T12:14:32.750 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-13T12:14:32.750 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T12:14:33.189 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672976 -lt 42949672976 2024-09-13T12:14:33.189 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:14:33.189 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509449 2024-09-13T12:14:33.189 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:14:33.190 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T12:14:33.191 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509449 2024-09-13T12:14:33.191 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:14:33.192 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509449 2024-09-13T12:14:33.192 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.2 seq 64424509449 2024-09-13T12:14:33.193 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-09-13T12:14:33.193 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T12:14:33.640 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509449 -lt 64424509449 2024-09-13T12:14:33.640 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2024-09-13T12:14:33.640 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T12:14:33.640 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T12:14:33.640 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T12:14:33.640 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T12:14:33.641 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T12:14:33.641 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T12:14:33.641 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T12:14:33.641 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T12:14:33.641 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T12:14:33.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T12:14:33.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T12:14:33.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T12:14:33.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T12:14:33.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T12:14:33.734 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T12:14:34.183 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T12:14:34.183 INFO:tasks.workunit.client.0.smithi032.stderr:1 2024-09-13T12:14:34.183 INFO:tasks.workunit.client.0.smithi032.stderr:2' 2024-09-13T12:14:34.183 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T12:14:34.183 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:14:34.183 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T12:14:34.307 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836510 2024-09-13T12:14:34.307 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836510 2024-09-13T12:14:34.308 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510' 2024-09-13T12:14:34.308 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:14:34.308 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T12:14:34.442 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672979 2024-09-13T12:14:34.442 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672979 2024-09-13T12:14:34.442 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-42949672979' 2024-09-13T12:14:34.442 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:14:34.442 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T12:14:34.565 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509452 2024-09-13T12:14:34.565 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509452 2024-09-13T12:14:34.566 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-42949672979 2-64424509452' 2024-09-13T12:14:34.566 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:14:34.566 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836510 2024-09-13T12:14:34.566 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:14:34.567 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T12:14:34.567 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836510 2024-09-13T12:14:34.567 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:14:34.568 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836510 2024-09-13T12:14:34.569 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.0 seq 21474836510 2024-09-13T12:14:34.569 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-09-13T12:14:34.569 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:14:34.975 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836508 -lt 21474836510 2024-09-13T12:14:34.976 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T12:14:35.977 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T12:14:35.978 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:14:36.444 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836511 -lt 21474836510 2024-09-13T12:14:36.445 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:14:36.445 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672979 2024-09-13T12:14:36.445 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:14:36.446 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T12:14:36.446 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672979 2024-09-13T12:14:36.446 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:14:36.447 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672979 2024-09-13T12:14:36.448 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.1 seq 42949672979 2024-09-13T12:14:36.448 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-13T12:14:36.448 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T12:14:36.887 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672979 -lt 42949672979 2024-09-13T12:14:36.887 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:14:36.887 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509452 2024-09-13T12:14:36.887 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:14:36.888 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T12:14:36.889 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509452 2024-09-13T12:14:36.889 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:14:36.890 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509452 2024-09-13T12:14:36.890 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-09-13T12:14:36.890 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.2 seq 64424509452 2024-09-13T12:14:36.890 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T12:14:37.318 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509452 -lt 64424509452 2024-09-13T12:14:37.319 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T12:14:37.319 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:14:37.319 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:14:37.890 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T12:14:37.890 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:14:37.890 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:14:37.890 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:14:37.890 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:14:37.890 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:14:37.890 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:14:37.890 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:14:38.297 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T12:14:38.298 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:14:38.298 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:14:38.298 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:14:38.856 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T12:14:38.856 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T12:14:38.856 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T12:14:38.857 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2024-09-13T12:14:38.857 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2024-09-13T12:14:39.264 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:14:39.276 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2024-09-13T12:14:39.276 INFO:tasks.workunit.client.0.smithi032.stdout:primary and soon to be divergent is 1 2024-09-13T12:14:39.280 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2024-09-13T12:14:39.280 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2024-09-13T12:14:39.665 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:14:39.665 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T12:14:24.859114+0000 0'0 29:17 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-13T12:14:22.777057+0000 0'0 2024-09-13T12:14:22.777057+0000 0 0 periodic scrub scheduled @ 2024-09-14T16:41:58.908543+0000 0 0 2024-09-13T12:14:39.665 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:14:39.665 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:14:39.666 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:14:39.677 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2024-09-13T12:14:39.677 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-09-13T12:14:39.677 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2024-09-13T12:14:39.677 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2024-09-13T12:14:39.678 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-09-13T12:14:39.678 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2024-09-13T12:14:39.678 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2024-09-13T12:14:39.678 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-09-13T12:14:39.678 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2024-09-13T12:14:39.678 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2024-09-13T12:14:39.678 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2024-09-13T12:14:39.678 INFO:tasks.workunit.client.0.smithi032.stdout:writing initial objects 2024-09-13T12:14:39.679 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2024-09-13T12:14:39.679 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:39.679 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2024-09-13T12:14:39.823 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:39.823 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2024-09-13T12:14:40.066 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:40.066 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2024-09-13T12:14:40.334 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:40.334 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2024-09-13T12:14:40.586 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:40.586 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2024-09-13T12:14:40.804 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:40.804 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2024-09-13T12:14:41.014 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:41.014 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2024-09-13T12:14:41.207 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:41.207 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2024-09-13T12:14:41.442 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:41.442 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2024-09-13T12:14:41.651 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:41.652 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2024-09-13T12:14:41.836 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:41.836 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2024-09-13T12:14:42.088 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:42.088 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2024-09-13T12:14:42.322 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:42.322 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2024-09-13T12:14:42.557 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:42.557 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2024-09-13T12:14:42.808 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:42.808 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2024-09-13T12:14:43.018 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:43.018 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2024-09-13T12:14:43.253 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:43.253 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2024-09-13T12:14:43.487 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:43.487 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2024-09-13T12:14:43.705 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:43.705 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2024-09-13T12:14:43.915 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:43.915 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2024-09-13T12:14:44.108 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:44.108 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2024-09-13T12:14:44.317 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:44.318 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2024-09-13T12:14:44.579 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:44.579 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2024-09-13T12:14:44.797 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:44.797 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2024-09-13T12:14:45.051 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:45.051 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2024-09-13T12:14:45.258 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:45.258 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2024-09-13T12:14:45.493 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:45.493 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2024-09-13T12:14:45.719 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:45.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2024-09-13T12:14:45.929 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:45.929 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2024-09-13T12:14:46.172 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:46.173 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2024-09-13T12:14:46.399 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:46.399 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2024-09-13T12:14:46.625 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:46.625 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2024-09-13T12:14:46.859 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:46.859 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2024-09-13T12:14:47.086 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:47.086 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2024-09-13T12:14:47.281 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:47.281 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2024-09-13T12:14:47.514 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:47.514 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2024-09-13T12:14:47.724 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:47.724 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2024-09-13T12:14:47.958 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:47.959 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2024-09-13T12:14:48.176 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:48.176 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2024-09-13T12:14:48.428 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:48.429 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2024-09-13T12:14:48.620 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:48.620 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2024-09-13T12:14:48.947 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:48.947 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2024-09-13T12:14:49.140 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:49.141 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2024-09-13T12:14:49.409 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:49.409 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2024-09-13T12:14:49.652 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:49.652 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2024-09-13T12:14:49.886 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:49.886 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2024-09-13T12:14:50.104 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:50.104 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2024-09-13T12:14:50.322 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:50.323 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2024-09-13T12:14:50.532 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:50.533 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2024-09-13T12:14:50.742 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:50.742 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2024-09-13T12:14:50.977 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:50.977 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2024-09-13T12:14:51.228 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:51.228 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2024-09-13T12:14:51.480 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:51.480 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2024-09-13T12:14:51.681 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:51.681 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2024-09-13T12:14:51.899 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:51.899 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2024-09-13T12:14:52.133 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:52.133 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2024-09-13T12:14:52.368 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:52.368 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2024-09-13T12:14:52.569 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:52.569 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2024-09-13T12:14:52.804 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:52.804 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2024-09-13T12:14:53.081 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:53.081 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2024-09-13T12:14:53.317 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:53.317 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2024-09-13T12:14:53.525 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:53.525 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2024-09-13T12:14:53.701 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:53.701 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2024-09-13T12:14:53.940 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:53.940 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2024-09-13T12:14:54.171 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:54.171 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2024-09-13T12:14:54.406 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:54.406 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2024-09-13T12:14:54.607 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:54.607 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2024-09-13T12:14:54.833 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:54.833 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2024-09-13T12:14:55.043 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:55.043 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2024-09-13T12:14:55.253 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:55.253 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2024-09-13T12:14:55.513 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:55.513 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2024-09-13T12:14:55.706 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:55.706 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2024-09-13T12:14:55.941 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:55.941 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2024-09-13T12:14:56.175 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:56.175 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2024-09-13T12:14:56.393 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:56.393 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2024-09-13T12:14:56.636 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:56.636 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2024-09-13T12:14:56.821 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:56.821 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2024-09-13T12:14:57.030 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:57.031 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2024-09-13T12:14:57.257 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:57.257 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2024-09-13T12:14:57.483 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:57.484 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2024-09-13T12:14:57.710 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:57.710 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2024-09-13T12:14:57.886 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:57.886 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2024-09-13T12:14:58.112 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:58.113 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2024-09-13T12:14:58.364 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:58.364 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2024-09-13T12:14:58.632 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:58.633 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2024-09-13T12:14:58.883 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:58.884 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2024-09-13T12:14:59.126 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:59.126 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2024-09-13T12:14:59.378 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:59.378 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2024-09-13T12:14:59.621 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:59.621 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2024-09-13T12:14:59.831 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:14:59.831 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2024-09-13T12:15:00.041 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:15:00.041 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2024-09-13T12:15:00.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:15:00.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2024-09-13T12:15:00.460 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:15:00.460 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2024-09-13T12:15:00.686 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:15:00.686 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2024-09-13T12:15:00.904 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:15:00.904 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2024-09-13T12:15:01.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:15:01.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2024-09-13T12:15:01.382 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:15:01.382 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2024-09-13T12:15:01.617 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:15:01.617 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2024-09-13T12:15:01.853 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:15:01.853 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-09-13T12:15:02.095 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-13T12:15:02.095 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-09-13T12:15:02.288 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-13T12:15:02.288 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2024-09-13T12:15:02.288 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T12:15:02.288 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T12:15:02.288 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T12:15:02.288 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-13T12:15:02.289 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T12:15:02.289 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T12:15:02.289 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T12:15:02.289 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T12:15:02.289 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T12:15:02.354 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-13T12:15:02.354 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T12:15:02.354 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T12:15:02.354 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T12:15:02.354 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T12:15:02.354 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T12:15:02.808 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T12:15:02.808 INFO:tasks.workunit.client.0.smithi032.stderr:1 2024-09-13T12:15:02.808 INFO:tasks.workunit.client.0.smithi032.stderr:2' 2024-09-13T12:15:02.808 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T12:15:02.808 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:15:02.808 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T12:15:02.933 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836518 2024-09-13T12:15:02.933 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836518 2024-09-13T12:15:02.933 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836518' 2024-09-13T12:15:02.933 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:15:02.933 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T12:15:03.057 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672987 2024-09-13T12:15:03.058 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672987 2024-09-13T12:15:03.058 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836518 1-42949672987' 2024-09-13T12:15:03.058 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:15:03.058 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T12:15:03.182 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509459 2024-09-13T12:15:03.182 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509459 2024-09-13T12:15:03.182 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836518 1-42949672987 2-64424509459' 2024-09-13T12:15:03.182 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:15:03.182 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836518 2024-09-13T12:15:03.182 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:15:03.183 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T12:15:03.184 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836518 2024-09-13T12:15:03.184 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:15:03.185 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836518 2024-09-13T12:15:03.185 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.0 seq 21474836518 2024-09-13T12:15:03.185 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2024-09-13T12:15:03.185 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:15:03.625 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836516 -lt 21474836518 2024-09-13T12:15:03.626 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T12:15:04.626 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T12:15:04.627 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:15:05.085 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836518 -lt 21474836518 2024-09-13T12:15:05.085 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:15:05.085 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672987 2024-09-13T12:15:05.086 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:15:05.086 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T12:15:05.087 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672987 2024-09-13T12:15:05.087 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:15:05.088 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672987 2024-09-13T12:15:05.088 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-09-13T12:15:05.088 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.1 seq 42949672987 2024-09-13T12:15:05.088 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T12:15:05.537 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672987 -lt 42949672987 2024-09-13T12:15:05.537 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:15:05.538 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509459 2024-09-13T12:15:05.538 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:15:05.538 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T12:15:05.539 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509459 2024-09-13T12:15:05.539 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:15:05.540 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509459 2024-09-13T12:15:05.540 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.2 seq 64424509459 2024-09-13T12:15:05.541 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-09-13T12:15:05.541 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T12:15:05.953 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509460 -lt 64424509459 2024-09-13T12:15:05.953 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T12:15:05.953 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:15:05.953 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:15:06.513 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T12:15:06.513 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:15:06.514 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:15:06.514 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:15:06.514 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:15:06.514 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:15:06.514 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:15:06.514 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:15:06.941 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T12:15:06.942 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:15:06.942 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:15:06.942 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:15:07.503 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T12:15:07.503 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T12:15:07.503 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T12:15:07.503 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2024-09-13T12:15:07.503 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=test 2024-09-13T12:15:07.503 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=existing_1 2024-09-13T12:15:07.504 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map test existing_1 2024-09-13T12:15:07.504 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T12:15:07.992 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2024-09-13T12:15:07.992 INFO:tasks.workunit.client.0.smithi032.stdout:blackholing osds 0 2 2024-09-13T12:15:07.993 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2024-09-13T12:15:07.993 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2024-09-13T12:15:08.390 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:15:08.390 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 100 0 0 0 0 48300 0 0 16 84 16 active+clean 2024-09-13T12:14:24.859114+0000 29'100 29:117 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-13T12:14:22.777057+0000 0'0 2024-09-13T12:14:22.777057+0000 0 0 periodic scrub scheduled @ 2024-09-14T16:41:58.908543+0000 0 0 2024-09-13T12:15:08.390 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:15:08.390 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:15:08.391 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:15:08.402 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-09-13T12:15:08.403 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2024-09-13T12:15:08.403 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-13T12:15:08.403 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-13T12:15:08.404 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:15:08.404 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:15:08.404 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:15:08.404 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-osd.0.asok 2024-09-13T12:15:08.404 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-09-13T12:15:08.404 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.33242/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-13T12:15:08.483 INFO:tasks.workunit.client.0.smithi032.stdout:{ 2024-09-13T12:15:08.483 INFO:tasks.workunit.client.0.smithi032.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-13T12:15:08.484 INFO:tasks.workunit.client.0.smithi032.stdout:} 2024-09-13T12:15:08.490 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-09-13T12:15:08.491 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2024-09-13T12:15:08.491 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-13T12:15:08.491 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-13T12:15:08.492 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:15:08.492 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:15:08.492 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:15:08.492 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-osd.2.asok 2024-09-13T12:15:08.492 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-09-13T12:15:08.492 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.33242/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-13T12:15:08.573 INFO:tasks.workunit.client.0.smithi032.stdout:{ 2024-09-13T12:15:08.573 INFO:tasks.workunit.client.0.smithi032.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-13T12:15:08.573 INFO:tasks.workunit.client.0.smithi032.stdout:} 2024-09-13T12:15:08.580 INFO:tasks.workunit.client.0.smithi032.stdout:writing divergent object 2024-09-13T12:15:08.581 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2024-09-13T12:15:08.581 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2024-09-13T12:15:08.581 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2024-09-13T12:15:09.582 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2024-09-13T12:15:09.582 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-09-13T12:15:09.809 INFO:tasks.workunit.client.0.smithi032.stdout:created pool test snap snap1 2024-09-13T12:15:09.812 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2024-09-13T12:15:09.812 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2024-09-13T12:15:09.814 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-09-13T12:15:19.814 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2024-09-13T12:15:19.818 INFO:tasks.workunit.client.0.smithi032.stderr:rados: no process found 2024-09-13T12:15:19.818 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2024-09-13T12:15:19.819 INFO:tasks.workunit.client.0.smithi032.stdout:killing all the osds 2024-09-13T12:15:19.819 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2024-09-13T12:15:20.218 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:15:20.218 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 101 0 0 0 0 48783 0 0 14 90 14 active+clean 2024-09-13T12:14:24.859114+0000 30'104 30:123 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-13T12:14:22.777057+0000 0'0 2024-09-13T12:14:22.777057+0000 0 0 periodic scrub scheduled @ 2024-09-14T23:03:45.459826+0000 0 0 2024-09-13T12:15:20.218 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:15:20.219 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:15:20.219 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:15:20.229 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2024-09-13T12:15:20.230 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:15:20.230 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:15:20.230 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:15:20.231 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:15:20.231 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:15:20.337 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:15:20.337 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-09-13T12:15:20.337 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2024-09-13T12:15:20.862 INFO:tasks.workunit.client.0.smithi032.stderr:osd.0 is already down. 2024-09-13T12:15:20.874 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-09-13T12:15:20.874 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2024-09-13T12:15:21.436 INFO:tasks.workunit.client.0.smithi032.stderr:osd.1 is already down. 2024-09-13T12:15:21.448 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-09-13T12:15:21.448 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2024-09-13T12:15:21.975 INFO:tasks.workunit.client.0.smithi032.stderr:osd.2 is already down. 2024-09-13T12:15:21.987 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-09-13T12:15:21.987 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2024-09-13T12:15:22.544 INFO:tasks.workunit.client.0.smithi032.stderr:osd.0 is already out. 2024-09-13T12:15:22.556 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-09-13T12:15:22.556 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2024-09-13T12:15:23.073 INFO:tasks.workunit.client.0.smithi032.stderr:osd.2 is already out. 2024-09-13T12:15:23.085 INFO:tasks.workunit.client.0.smithi032.stdout:bringing up non_divergent 0 2 2024-09-13T12:15:23.085 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2024-09-13T12:15:23.085 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2024-09-13T12:15:23.475 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:15:23.475 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 101 0 0 0 0 48783 0 0 14 90 14 stale+active+clean 2024-09-13T12:14:24.859114+0000 30'104 30:123 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-13T12:14:22.777057+0000 0'0 2024-09-13T12:14:22.777057+0000 0 0 periodic scrub scheduled @ 2024-09-14T23:03:45.459826+0000 0 0 2024-09-13T12:15:23.475 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:15:23.475 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:15:23.475 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:15:23.489 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-09-13T12:15:23.489 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-09-13T12:15:23.489 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-13T12:15:23.489 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:15:23.490 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T12:15:23.490 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:15:23.490 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/0 2024-09-13T12:15:23.490 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T12:15:23.490 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:15:23.490 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:15:23.490 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:15:23.490 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-13T12:15:23.490 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-13T12:15:23.490 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:15:23.490 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:15:23.490 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T12:15:23.490 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:15:23.491 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:15:23.491 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:15:23.491 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:15:23.491 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:15:23.491 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:15:23.491 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:15:23.491 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:15:23.491 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:15:23.491 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T12:15:23.492 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T12:15:23.492 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:15:23.492 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:15:23.492 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:15:23.492 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:15:23.492 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:15:23.492 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:15:23.492 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/0 2024-09-13T12:15:23.492 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.0 2024-09-13T12:15:23.493 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T12:15:23.493 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:15:23.493 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/0/whoami 2024-09-13T12:15:23.494 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T12:15:23.494 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:15:23.494 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:15:23.494 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:15:23.525 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:15:23.523+0000 7f2ece016640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:15:23.527 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:15:23.525+0000 7f2ece016640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:15:23.528 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:15:23.526+0000 7f2ece016640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:15:23.933 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T12:15:23.934 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:15:23.934 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T12:15:23.934 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:15:23.934 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:15:23.934 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:15:23.934 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:15:23.934 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:15:23.935 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:15:23.935 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:15:24.343 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:15:24.623 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:15:24.622+0000 7f2ece016640 -1 Falling back to public interface 2024-09-13T12:15:25.345 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:15:25.345 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:15:25.345 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:15:25.345 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:15:25.346 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:15:25.346 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:15:25.802 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:15:26.803 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:15:26.803 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:15:26.803 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:15:26.804 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:15:26.804 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:15:26.804 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:15:27.228 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:15:27.797 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:15:27.795+0000 7f2ece016640 -1 osd.0 29 log_to_monitors true 2024-09-13T12:15:28.229 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:15:28.229 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:15:28.229 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:15:28.231 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:15:28.231 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:15:28.231 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:15:28.643 INFO:tasks.workunit.client.0.smithi032.stdout:osd.0 up out weight 0 up_from 37 up_thru 0 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/4196072543,v1:127.0.0.1:6803/4196072543] [v2:127.0.0.1:6804/4196072543,v1:127.0.0.1:6805/4196072543] exists,up 92bce19c-5a5e-4360-a088-1bf366fc2ecf 2024-09-13T12:15:28.646 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:15:28.646 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:15:28.646 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:15:28.646 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-09-13T12:15:28.646 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-09-13T12:15:28.646 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-13T12:15:28.646 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:15:28.647 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-13T12:15:28.647 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:15:28.647 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/2 2024-09-13T12:15:28.647 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T12:15:28.647 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:15:28.647 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:15:28.647 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:15:28.647 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-13T12:15:28.647 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-13T12:15:28.647 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:15:28.647 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:15:28.647 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T12:15:28.647 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:15:28.648 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:15:28.648 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:15:28.648 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:15:28.648 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:15:28.648 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:15:28.648 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:15:28.648 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:15:28.648 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:15:28.648 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T12:15:28.648 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T12:15:28.648 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:15:28.649 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:15:28.649 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:15:28.649 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:15:28.649 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:15:28.649 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:15:28.649 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/2 2024-09-13T12:15:28.649 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-13T12:15:28.649 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.2 2024-09-13T12:15:28.650 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:15:28.650 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/2/whoami 2024-09-13T12:15:28.650 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-13T12:15:28.650 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:15:28.650 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:15:28.651 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:15:28.682 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:15:28.679+0000 7f900181a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:15:28.683 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:15:28.681+0000 7f900181a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:15:28.684 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:15:28.683+0000 7f900181a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:15:29.088 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-13T12:15:29.088 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:15:29.088 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T12:15:29.088 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:15:29.089 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:15:29.089 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:15:29.089 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:15:29.089 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:15:29.090 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:15:29.090 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:15:29.538 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:15:30.037 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:15:30.036+0000 7f900181a640 -1 Falling back to public interface 2024-09-13T12:15:30.539 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:15:30.540 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:15:30.540 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:15:30.540 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:15:30.540 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:15:30.541 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:15:30.989 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:15:31.991 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:15:31.991 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:15:31.991 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:15:31.991 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:15:31.992 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:15:31.992 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:15:32.466 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:15:33.467 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:15:33.468 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:15:33.468 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:15:33.469 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:15:33.469 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:15:33.469 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:15:33.476 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:15:33.475+0000 7f900181a640 -1 osd.2 29 log_to_monitors true 2024-09-13T12:15:33.899 INFO:tasks.workunit.client.0.smithi032.stdout:osd.2 up out weight 0 up_from 39 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/3815047400,v1:127.0.0.1:6811/3815047400] [v2:127.0.0.1:6812/3815047400,v1:127.0.0.1:6813/3815047400] exists,up 575c91a2-dd41-4141-bbec-f3711b8a1a2e 2024-09-13T12:15:33.900 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:15:33.900 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:15:33.900 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:15:33.900 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-09-13T12:15:33.900 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2024-09-13T12:15:34.472 INFO:tasks.workunit.client.0.smithi032.stderr:osd.0 is already in. 2024-09-13T12:15:34.484 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-09-13T12:15:34.484 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2024-09-13T12:15:35.021 INFO:tasks.workunit.client.0.smithi032.stderr:osd.2 is already in. 2024-09-13T12:15:35.031 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2024-09-13T12:15:40.033 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2024-09-13T12:15:40.034 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2024-09-13T12:15:40.034 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2024-09-13T12:15:40.034 INFO:tasks.workunit.client.0.smithi032.stdout:writing non-divergent object existing_4 2024-09-13T12:15:40.034 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2024-09-13T12:15:40.439 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:15:40.439 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 100 0 100 0 0 48300 0 0 16 84 16 active+undersized+degraded 2024-09-13T12:15:35.391757+0000 29'100 42:127 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-13T12:14:22.777057+0000 0'0 2024-09-13T12:14:22.777057+0000 0 0 periodic scrub scheduled @ 2024-09-14T20:15:05.517609+0000 0 0 2024-09-13T12:15:40.439 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:15:40.439 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:15:40.440 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:15:40.451 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2024-09-13T12:15:40.547 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-13T12:15:40.547 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2024-09-13T12:15:40.547 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T12:15:40.548 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T12:15:40.548 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T12:15:40.548 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-13T12:15:40.548 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T12:15:40.549 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T12:15:40.549 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T12:15:40.549 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T12:15:40.549 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T12:15:40.605 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-13T12:15:40.605 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T12:15:40.605 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T12:15:40.605 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T12:15:40.605 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T12:15:40.605 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T12:15:41.042 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T12:15:41.042 INFO:tasks.workunit.client.0.smithi032.stderr:1 2024-09-13T12:15:41.042 INFO:tasks.workunit.client.0.smithi032.stderr:2' 2024-09-13T12:15:41.042 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T12:15:41.043 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:15:41.043 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T12:15:41.167 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=158913789957 2024-09-13T12:15:41.168 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 158913789957 2024-09-13T12:15:41.168 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-158913789957' 2024-09-13T12:15:41.168 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:15:41.168 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T12:15:41.282 INFO:tasks.workunit.client.0.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-13T12:15:41.284 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-13T12:15:41.284 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-13T12:15:41.285 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-13T12:15:41.285 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:15:41.285 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T12:15:41.410 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=167503724548 2024-09-13T12:15:41.410 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 167503724548 2024-09-13T12:15:41.410 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-158913789957 2-167503724548' 2024-09-13T12:15:41.411 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:15:41.411 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-158913789957 2024-09-13T12:15:41.411 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:15:41.412 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T12:15:41.412 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-158913789957 2024-09-13T12:15:41.412 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:15:41.413 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=158913789957 2024-09-13T12:15:41.413 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 158913789957' 2024-09-13T12:15:41.414 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.0 seq 158913789957 2024-09-13T12:15:41.415 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:15:41.868 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 158913789955 -lt 158913789957 2024-09-13T12:15:41.868 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T12:15:42.869 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T12:15:42.870 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:15:43.296 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 158913789957 -lt 158913789957 2024-09-13T12:15:43.297 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:15:43.297 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-167503724548 2024-09-13T12:15:43.297 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:15:43.298 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T12:15:43.298 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-167503724548 2024-09-13T12:15:43.298 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:15:43.299 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=167503724548 2024-09-13T12:15:43.300 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 167503724548' 2024-09-13T12:15:43.300 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.2 seq 167503724548 2024-09-13T12:15:43.300 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T12:15:43.771 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 167503724548 -lt 167503724548 2024-09-13T12:15:43.771 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T12:15:43.771 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:15:43.772 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:15:44.330 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T12:15:44.330 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:15:44.330 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:15:44.331 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:15:44.331 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:15:44.331 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:15:44.331 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:15:44.331 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:15:44.730 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:15:44.731 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:15:44.731 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:15:44.731 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:15:45.293 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:15:45.293 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-13T12:15:45.293 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T12:15:45.294 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-13T12:15:45.294 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:15:45.294 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T12:15:45.391 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:15:45.392 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:15:45.392 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:15:45.392 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:15:45.392 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:15:45.392 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:15:45.392 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:15:45.392 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:15:45.795 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:15:45.796 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:15:45.796 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:15:45.796 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:15:46.365 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:15:46.366 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:15:46.366 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:15:46.366 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:15:46.366 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:15:46.366 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:15:46.366 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:15:46.366 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:15:46.366 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:15:46.973 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:15:46.974 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:15:46.974 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-13T12:15:46.974 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:15:46.974 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T12:15:47.175 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:15:47.175 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:15:47.175 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:15:47.175 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:15:47.175 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:15:47.175 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:15:47.175 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:15:47.175 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:15:47.574 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:15:47.574 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:15:47.575 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:15:47.575 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:15:48.130 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:15:48.130 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:15:48.130 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:15:48.130 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:15:48.130 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:15:48.130 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:15:48.130 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:15:48.131 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:15:48.131 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:15:48.710 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:15:48.710 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:15:48.710 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-13T12:15:48.710 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:15:48.710 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-13T12:15:49.112 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:15:49.112 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:15:49.112 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:15:49.112 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:15:49.112 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:15:49.112 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:15:49.112 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:15:49.112 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:15:49.521 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:15:49.521 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:15:49.522 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:15:49.522 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:15:50.081 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:15:50.081 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:15:50.081 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:15:50.082 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:15:50.082 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:15:50.082 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:15:50.082 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:15:50.082 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:15:50.082 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:15:50.645 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:15:50.645 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:15:50.646 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-13T12:15:50.646 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:15:50.646 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-13T12:15:51.446 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:15:51.446 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:15:51.447 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:15:51.447 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:15:51.447 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:15:51.448 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:15:51.448 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:15:51.448 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:15:51.856 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:15:51.856 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:15:51.857 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:15:51.857 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:15:52.443 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:15:52.444 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:15:52.444 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:15:52.444 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:15:52.444 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:15:52.444 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:15:52.444 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:15:52.444 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:15:52.444 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:15:53.006 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:15:53.006 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:15:53.006 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-13T12:15:53.006 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:15:53.006 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-13T12:15:54.606 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:15:54.607 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:15:54.607 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:15:54.607 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:15:54.608 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:15:54.608 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:15:54.608 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:15:54.608 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:15:55.020 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:15:55.020 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:15:55.021 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:15:55.021 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:15:55.578 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:15:55.578 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:15:55.579 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:15:55.579 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:15:55.579 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:15:55.579 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:15:55.579 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:15:55.579 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:15:55.579 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:15:56.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:15:56.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:15:56.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-13T12:15:56.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:15:56.139 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-13T12:15:59.340 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:15:59.340 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:15:59.341 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:15:59.341 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:15:59.341 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:15:59.342 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:15:59.342 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:15:59.342 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:15:59.751 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:15:59.752 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:15:59.752 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:15:59.752 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:16:00.311 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:16:00.311 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:16:00.311 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:16:00.311 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:16:00.311 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:16:00.311 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:16:00.311 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:16:00.311 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:16:00.311 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:16:00.879 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:16:00.879 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:16:00.879 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-13T12:16:00.879 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:16:00.879 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-13T12:16:07.281 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:16:07.282 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:16:07.282 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:16:07.282 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:16:07.282 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:16:07.282 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:16:07.282 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:16:07.282 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:16:07.697 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:16:07.697 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:16:07.697 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:16:07.698 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:16:08.255 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:16:08.255 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:16:08.255 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:16:08.255 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:16:08.255 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:16:08.255 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:16:08.255 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:16:08.256 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:16:08.256 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:16:08.818 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:16:08.818 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:16:08.818 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-13T12:16:08.818 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:16:08.818 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-13T12:16:16.119 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:16:16.119 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:16:16.120 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:16:16.120 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:16:16.120 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:16:16.120 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:16:16.120 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:16:16.121 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:16:16.531 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:16:16.532 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:16:16.532 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:16:16.532 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:16:17.094 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:16:17.094 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:16:17.094 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:16:17.094 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:16:17.094 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:16:17.094 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:16:17.094 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:16:17.094 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:16:17.094 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:16:17.665 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:16:17.665 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:16:17.665 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-13T12:16:17.665 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-13T12:16:18.223 INFO:tasks.workunit.client.0.smithi032.stdout:{ 2024-09-13T12:16:18.223 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_fingerprint": "6ab5cc67-92b9-437c-9d83-cc24da6da1c9", 2024-09-13T12:16:18.223 INFO:tasks.workunit.client.0.smithi032.stdout: "version": "19.3.0-4697-gd58f7306", 2024-09-13T12:16:18.223 INFO:tasks.workunit.client.0.smithi032.stdout: "commit": "d58f73069a8157a9ace12a0247ee3e8497707ea9", 2024-09-13T12:16:18.223 INFO:tasks.workunit.client.0.smithi032.stdout: "timestamp": "2024-09-13T12:16:18.221764+0000", 2024-09-13T12:16:18.223 INFO:tasks.workunit.client.0.smithi032.stdout: "tag": "", 2024-09-13T12:16:18.223 INFO:tasks.workunit.client.0.smithi032.stdout: "health": { 2024-09-13T12:16:18.223 INFO:tasks.workunit.client.0.smithi032.stdout: "status": "HEALTH_WARN", 2024-09-13T12:16:18.223 INFO:tasks.workunit.client.0.smithi032.stdout: "checks": { 2024-09-13T12:16:18.223 INFO:tasks.workunit.client.0.smithi032.stdout: "OSDMAP_FLAGS": { 2024-09-13T12:16:18.224 INFO:tasks.workunit.client.0.smithi032.stdout: "severity": "HEALTH_WARN", 2024-09-13T12:16:18.224 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": { 2024-09-13T12:16:18.224 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-13T12:16:18.224 INFO:tasks.workunit.client.0.smithi032.stdout: "count": 17 2024-09-13T12:16:18.224 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.224 INFO:tasks.workunit.client.0.smithi032.stdout: "detail": [], 2024-09-13T12:16:18.224 INFO:tasks.workunit.client.0.smithi032.stdout: "muted": false 2024-09-13T12:16:18.224 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.224 INFO:tasks.workunit.client.0.smithi032.stdout: "OSD_DOWN": { 2024-09-13T12:16:18.224 INFO:tasks.workunit.client.0.smithi032.stdout: "severity": "HEALTH_WARN", 2024-09-13T12:16:18.224 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": { 2024-09-13T12:16:18.224 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "1 osds down", 2024-09-13T12:16:18.224 INFO:tasks.workunit.client.0.smithi032.stdout: "count": 1 2024-09-13T12:16:18.225 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.225 INFO:tasks.workunit.client.0.smithi032.stdout: "detail": [ 2024-09-13T12:16:18.225 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.225 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "osd.1 (root=default,host=smithi032) is down" 2024-09-13T12:16:18.225 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.225 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:16:18.225 INFO:tasks.workunit.client.0.smithi032.stdout: "muted": false 2024-09-13T12:16:18.225 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.225 INFO:tasks.workunit.client.0.smithi032.stdout: "PG_DEGRADED": { 2024-09-13T12:16:18.225 INFO:tasks.workunit.client.0.smithi032.stdout: "severity": "HEALTH_WARN", 2024-09-13T12:16:18.225 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": { 2024-09-13T12:16:18.225 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-09-13T12:16:18.225 INFO:tasks.workunit.client.0.smithi032.stdout: "count": 1 2024-09-13T12:16:18.226 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.226 INFO:tasks.workunit.client.0.smithi032.stdout: "detail": [ 2024-09-13T12:16:18.226 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.226 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-09-13T12:16:18.226 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.226 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:16:18.226 INFO:tasks.workunit.client.0.smithi032.stdout: "muted": false 2024-09-13T12:16:18.226 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.226 INFO:tasks.workunit.client.0.smithi032.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-13T12:16:18.226 INFO:tasks.workunit.client.0.smithi032.stdout: "severity": "HEALTH_WARN", 2024-09-13T12:16:18.226 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": { 2024-09-13T12:16:18.226 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-13T12:16:18.226 INFO:tasks.workunit.client.0.smithi032.stdout: "count": 1 2024-09-13T12:16:18.226 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.226 INFO:tasks.workunit.client.0.smithi032.stdout: "detail": [ 2024-09-13T12:16:18.227 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.227 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "application not enabled on pool 'test'" 2024-09-13T12:16:18.227 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.227 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.227 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-13T12:16:18.227 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.227 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:16:18.227 INFO:tasks.workunit.client.0.smithi032.stdout: "muted": false 2024-09-13T12:16:18.227 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.227 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.227 INFO:tasks.workunit.client.0.smithi032.stdout: "mutes": [] 2024-09-13T12:16:18.227 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.227 INFO:tasks.workunit.client.0.smithi032.stdout: "monmap_first_committed": 1, 2024-09-13T12:16:18.227 INFO:tasks.workunit.client.0.smithi032.stdout: "monmap_last_committed": 1, 2024-09-13T12:16:18.228 INFO:tasks.workunit.client.0.smithi032.stdout: "monmap": { 2024-09-13T12:16:18.228 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 1, 2024-09-13T12:16:18.228 INFO:tasks.workunit.client.0.smithi032.stdout: "fsid": "e12dfbc3-9bf5-45db-a6f0-f25db93ee351", 2024-09-13T12:16:18.228 INFO:tasks.workunit.client.0.smithi032.stdout: "modified": "2024-09-13T12:12:30.364120Z", 2024-09-13T12:16:18.228 INFO:tasks.workunit.client.0.smithi032.stdout: "created": "2024-09-13T12:12:30.364120Z", 2024-09-13T12:16:18.228 INFO:tasks.workunit.client.0.smithi032.stdout: "min_mon_release": 19, 2024-09-13T12:16:18.228 INFO:tasks.workunit.client.0.smithi032.stdout: "min_mon_release_name": "squid", 2024-09-13T12:16:18.228 INFO:tasks.workunit.client.0.smithi032.stdout: "election_strategy": 1, 2024-09-13T12:16:18.228 INFO:tasks.workunit.client.0.smithi032.stdout: "disallowed_leaders: ": "", 2024-09-13T12:16:18.228 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_mode": false, 2024-09-13T12:16:18.228 INFO:tasks.workunit.client.0.smithi032.stdout: "tiebreaker_mon": "", 2024-09-13T12:16:18.228 INFO:tasks.workunit.client.0.smithi032.stdout: "removed_ranks: ": "", 2024-09-13T12:16:18.228 INFO:tasks.workunit.client.0.smithi032.stdout: "features": { 2024-09-13T12:16:18.229 INFO:tasks.workunit.client.0.smithi032.stdout: "persistent": [ 2024-09-13T12:16:18.229 INFO:tasks.workunit.client.0.smithi032.stdout: "kraken", 2024-09-13T12:16:18.229 INFO:tasks.workunit.client.0.smithi032.stdout: "luminous", 2024-09-13T12:16:18.229 INFO:tasks.workunit.client.0.smithi032.stdout: "mimic", 2024-09-13T12:16:18.229 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap-prune", 2024-09-13T12:16:18.229 INFO:tasks.workunit.client.0.smithi032.stdout: "nautilus", 2024-09-13T12:16:18.229 INFO:tasks.workunit.client.0.smithi032.stdout: "octopus", 2024-09-13T12:16:18.229 INFO:tasks.workunit.client.0.smithi032.stdout: "pacific", 2024-09-13T12:16:18.229 INFO:tasks.workunit.client.0.smithi032.stdout: "elector-pinging", 2024-09-13T12:16:18.229 INFO:tasks.workunit.client.0.smithi032.stdout: "quincy", 2024-09-13T12:16:18.229 INFO:tasks.workunit.client.0.smithi032.stdout: "reef", 2024-09-13T12:16:18.229 INFO:tasks.workunit.client.0.smithi032.stdout: "squid" 2024-09-13T12:16:18.229 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:16:18.230 INFO:tasks.workunit.client.0.smithi032.stdout: "optional": [] 2024-09-13T12:16:18.230 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.230 INFO:tasks.workunit.client.0.smithi032.stdout: "mons": [ 2024-09-13T12:16:18.230 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.230 INFO:tasks.workunit.client.0.smithi032.stdout: "rank": 0, 2024-09-13T12:16:18.230 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "a", 2024-09-13T12:16:18.230 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addrs": { 2024-09-13T12:16:18.230 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:16:18.230 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.230 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:16:18.230 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:7115", 2024-09-13T12:16:18.231 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 0 2024-09-13T12:16:18.231 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.231 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:16:18.231 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.231 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:7115/0", 2024-09-13T12:16:18.231 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-13T12:16:18.231 INFO:tasks.workunit.client.0.smithi032.stdout: "priority": 0, 2024-09-13T12:16:18.231 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 0, 2024-09-13T12:16:18.231 INFO:tasks.workunit.client.0.smithi032.stdout: "crush_location": "{}" 2024-09-13T12:16:18.231 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.231 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:16:18.231 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.231 INFO:tasks.workunit.client.0.smithi032.stdout: "quorum": [ 2024-09-13T12:16:18.231 INFO:tasks.workunit.client.0.smithi032.stdout: 0 2024-09-13T12:16:18.232 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:16:18.232 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap": { 2024-09-13T12:16:18.232 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 42, 2024-09-13T12:16:18.232 INFO:tasks.workunit.client.0.smithi032.stdout: "fsid": "e12dfbc3-9bf5-45db-a6f0-f25db93ee351", 2024-09-13T12:16:18.232 INFO:tasks.workunit.client.0.smithi032.stdout: "created": "2024-09-13T12:12:31.284734+0000", 2024-09-13T12:16:18.232 INFO:tasks.workunit.client.0.smithi032.stdout: "modified": "2024-09-13T12:15:35.099540+0000", 2024-09-13T12:16:18.232 INFO:tasks.workunit.client.0.smithi032.stdout: "last_up_change": "2024-09-13T12:15:33.675262+0000", 2024-09-13T12:16:18.232 INFO:tasks.workunit.client.0.smithi032.stdout: "last_in_change": "2024-09-13T12:15:34.847471+0000", 2024-09-13T12:16:18.232 INFO:tasks.workunit.client.0.smithi032.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T12:16:18.232 INFO:tasks.workunit.client.0.smithi032.stdout: "flags_num": 5800384, 2024-09-13T12:16:18.232 INFO:tasks.workunit.client.0.smithi032.stdout: "flags_set": [ 2024-09-13T12:16:18.232 INFO:tasks.workunit.client.0.smithi032.stdout: "nodown", 2024-09-13T12:16:18.232 INFO:tasks.workunit.client.0.smithi032.stdout: "noin", 2024-09-13T12:16:18.232 INFO:tasks.workunit.client.0.smithi032.stdout: "noout", 2024-09-13T12:16:18.233 INFO:tasks.workunit.client.0.smithi032.stdout: "pglog_hardlimit", 2024-09-13T12:16:18.233 INFO:tasks.workunit.client.0.smithi032.stdout: "purged_snapdirs", 2024-09-13T12:16:18.233 INFO:tasks.workunit.client.0.smithi032.stdout: "recovery_deletes", 2024-09-13T12:16:18.233 INFO:tasks.workunit.client.0.smithi032.stdout: "sortbitwise" 2024-09-13T12:16:18.233 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:16:18.233 INFO:tasks.workunit.client.0.smithi032.stdout: "crush_version": 10, 2024-09-13T12:16:18.233 INFO:tasks.workunit.client.0.smithi032.stdout: "full_ratio": 0.99000000953674316, 2024-09-13T12:16:18.233 INFO:tasks.workunit.client.0.smithi032.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-13T12:16:18.233 INFO:tasks.workunit.client.0.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T12:16:18.233 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_snapshot": "", 2024-09-13T12:16:18.233 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_max": 1, 2024-09-13T12:16:18.233 INFO:tasks.workunit.client.0.smithi032.stdout: "max_osd": 3, 2024-09-13T12:16:18.233 INFO:tasks.workunit.client.0.smithi032.stdout: "require_min_compat_client": "luminous", 2024-09-13T12:16:18.234 INFO:tasks.workunit.client.0.smithi032.stdout: "min_compat_client": "jewel", 2024-09-13T12:16:18.234 INFO:tasks.workunit.client.0.smithi032.stdout: "require_osd_release": "squid", 2024-09-13T12:16:18.234 INFO:tasks.workunit.client.0.smithi032.stdout: "allow_crimson": false, 2024-09-13T12:16:18.234 INFO:tasks.workunit.client.0.smithi032.stdout: "pools": [ 2024-09-13T12:16:18.234 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.234 INFO:tasks.workunit.client.0.smithi032.stdout: "pool": 1, 2024-09-13T12:16:18.234 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_name": "test", 2024-09-13T12:16:18.234 INFO:tasks.workunit.client.0.smithi032.stdout: "create_time": "2024-09-13T12:14:22.654401+0000", 2024-09-13T12:16:18.234 INFO:tasks.workunit.client.0.smithi032.stdout: "flags": 16389, 2024-09-13T12:16:18.234 INFO:tasks.workunit.client.0.smithi032.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-09-13T12:16:18.234 INFO:tasks.workunit.client.0.smithi032.stdout: "type": 3, 2024-09-13T12:16:18.234 INFO:tasks.workunit.client.0.smithi032.stdout: "size": 3, 2024-09-13T12:16:18.234 INFO:tasks.workunit.client.0.smithi032.stdout: "min_size": 2, 2024-09-13T12:16:18.235 INFO:tasks.workunit.client.0.smithi032.stdout: "crush_rule": 1, 2024-09-13T12:16:18.235 INFO:tasks.workunit.client.0.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-13T12:16:18.235 INFO:tasks.workunit.client.0.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-13T12:16:18.235 INFO:tasks.workunit.client.0.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T12:16:18.235 INFO:tasks.workunit.client.0.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T12:16:18.235 INFO:tasks.workunit.client.0.smithi032.stdout: "is_stretch_pool": false, 2024-09-13T12:16:18.235 INFO:tasks.workunit.client.0.smithi032.stdout: "object_hash": 2, 2024-09-13T12:16:18.235 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_autoscale_mode": "off", 2024-09-13T12:16:18.235 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_num": 1, 2024-09-13T12:16:18.235 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_placement_num": 1, 2024-09-13T12:16:18.235 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_placement_num_target": 1, 2024-09-13T12:16:18.235 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_num_target": 1, 2024-09-13T12:16:18.235 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_num_pending": 1, 2024-09-13T12:16:18.235 INFO:tasks.workunit.client.0.smithi032.stdout: "last_pg_merge_meta": { 2024-09-13T12:16:18.236 INFO:tasks.workunit.client.0.smithi032.stdout: "source_pgid": "0.0", 2024-09-13T12:16:18.236 INFO:tasks.workunit.client.0.smithi032.stdout: "ready_epoch": 0, 2024-09-13T12:16:18.236 INFO:tasks.workunit.client.0.smithi032.stdout: "last_epoch_started": 0, 2024-09-13T12:16:18.236 INFO:tasks.workunit.client.0.smithi032.stdout: "last_epoch_clean": 0, 2024-09-13T12:16:18.236 INFO:tasks.workunit.client.0.smithi032.stdout: "source_version": "0'0", 2024-09-13T12:16:18.236 INFO:tasks.workunit.client.0.smithi032.stdout: "target_version": "0'0" 2024-09-13T12:16:18.236 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.236 INFO:tasks.workunit.client.0.smithi032.stdout: "last_change": "30", 2024-09-13T12:16:18.236 INFO:tasks.workunit.client.0.smithi032.stdout: "last_force_op_resend": "0", 2024-09-13T12:16:18.236 INFO:tasks.workunit.client.0.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T12:16:18.236 INFO:tasks.workunit.client.0.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T12:16:18.237 INFO:tasks.workunit.client.0.smithi032.stdout: "auid": 0, 2024-09-13T12:16:18.237 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_mode": "pool", 2024-09-13T12:16:18.237 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_seq": 1, 2024-09-13T12:16:18.237 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_epoch": 30, 2024-09-13T12:16:18.237 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_snaps": [ 2024-09-13T12:16:18.237 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.237 INFO:tasks.workunit.client.0.smithi032.stdout: "snapid": 1, 2024-09-13T12:16:18.237 INFO:tasks.workunit.client.0.smithi032.stdout: "stamp": "2024-09-13T12:15:09.622310+0000", 2024-09-13T12:16:18.237 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "snap1" 2024-09-13T12:16:18.237 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.237 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:16:18.237 INFO:tasks.workunit.client.0.smithi032.stdout: "removed_snaps": "[]", 2024-09-13T12:16:18.238 INFO:tasks.workunit.client.0.smithi032.stdout: "quota_max_bytes": 0, 2024-09-13T12:16:18.238 INFO:tasks.workunit.client.0.smithi032.stdout: "quota_max_objects": 0, 2024-09-13T12:16:18.238 INFO:tasks.workunit.client.0.smithi032.stdout: "tiers": [], 2024-09-13T12:16:18.238 INFO:tasks.workunit.client.0.smithi032.stdout: "tier_of": -1, 2024-09-13T12:16:18.238 INFO:tasks.workunit.client.0.smithi032.stdout: "read_tier": -1, 2024-09-13T12:16:18.238 INFO:tasks.workunit.client.0.smithi032.stdout: "write_tier": -1, 2024-09-13T12:16:18.238 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_mode": "none", 2024-09-13T12:16:18.238 INFO:tasks.workunit.client.0.smithi032.stdout: "target_max_bytes": 0, 2024-09-13T12:16:18.238 INFO:tasks.workunit.client.0.smithi032.stdout: "target_max_objects": 0, 2024-09-13T12:16:18.238 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T12:16:18.238 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T12:16:18.238 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T12:16:18.238 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-13T12:16:18.238 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-13T12:16:18.239 INFO:tasks.workunit.client.0.smithi032.stdout: "erasure_code_profile": "myprofile", 2024-09-13T12:16:18.239 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_params": { 2024-09-13T12:16:18.239 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "none" 2024-09-13T12:16:18.239 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.239 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_period": 0, 2024-09-13T12:16:18.239 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_count": 0, 2024-09-13T12:16:18.239 INFO:tasks.workunit.client.0.smithi032.stdout: "use_gmt_hitset": true, 2024-09-13T12:16:18.239 INFO:tasks.workunit.client.0.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-13T12:16:18.239 INFO:tasks.workunit.client.0.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-13T12:16:18.239 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T12:16:18.239 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-13T12:16:18.239 INFO:tasks.workunit.client.0.smithi032.stdout: "grade_table": [], 2024-09-13T12:16:18.239 INFO:tasks.workunit.client.0.smithi032.stdout: "stripe_width": 8192, 2024-09-13T12:16:18.239 INFO:tasks.workunit.client.0.smithi032.stdout: "expected_num_objects": 0, 2024-09-13T12:16:18.240 INFO:tasks.workunit.client.0.smithi032.stdout: "fast_read": false, 2024-09-13T12:16:18.240 INFO:tasks.workunit.client.0.smithi032.stdout: "options": {}, 2024-09-13T12:16:18.240 INFO:tasks.workunit.client.0.smithi032.stdout: "application_metadata": {} 2024-09-13T12:16:18.240 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.240 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:16:18.240 INFO:tasks.workunit.client.0.smithi032.stdout: "osds": [ 2024-09-13T12:16:18.240 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.240 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 0, 2024-09-13T12:16:18.240 INFO:tasks.workunit.client.0.smithi032.stdout: "uuid": "92bce19c-5a5e-4360-a088-1bf366fc2ecf", 2024-09-13T12:16:18.240 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 1, 2024-09-13T12:16:18.240 INFO:tasks.workunit.client.0.smithi032.stdout: "in": 1, 2024-09-13T12:16:18.240 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 1, 2024-09-13T12:16:18.240 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_affinity": 1, 2024-09-13T12:16:18.240 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_begin": 5, 2024-09-13T12:16:18.241 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_end": 29, 2024-09-13T12:16:18.241 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 37, 2024-09-13T12:16:18.241 INFO:tasks.workunit.client.0.smithi032.stdout: "up_thru": 41, 2024-09-13T12:16:18.241 INFO:tasks.workunit.client.0.smithi032.stdout: "down_at": 31, 2024-09-13T12:16:18.241 INFO:tasks.workunit.client.0.smithi032.stdout: "lost_at": 0, 2024-09-13T12:16:18.241 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addrs": { 2024-09-13T12:16:18.241 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:16:18.241 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.241 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:16:18.241 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6802", 2024-09-13T12:16:18.241 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 4196072543 2024-09-13T12:16:18.241 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.241 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.241 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:16:18.242 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6803", 2024-09-13T12:16:18.242 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 4196072543 2024-09-13T12:16:18.242 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.242 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:16:18.242 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.242 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addrs": { 2024-09-13T12:16:18.242 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:16:18.242 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.242 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:16:18.242 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6804", 2024-09-13T12:16:18.242 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 4196072543 2024-09-13T12:16:18.242 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.242 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.242 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:16:18.243 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6805", 2024-09-13T12:16:18.243 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 4196072543 2024-09-13T12:16:18.243 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.243 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:16:18.243 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.243 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-13T12:16:18.243 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:16:18.243 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.243 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:16:18.243 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6808", 2024-09-13T12:16:18.243 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 4196072543 2024-09-13T12:16:18.243 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.243 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.243 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:16:18.244 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6809", 2024-09-13T12:16:18.244 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 4196072543 2024-09-13T12:16:18.244 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.244 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:16:18.244 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.244 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-13T12:16:18.244 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:16:18.244 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.244 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:16:18.244 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6806", 2024-09-13T12:16:18.244 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 4196072543 2024-09-13T12:16:18.244 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.244 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.244 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:16:18.244 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6807", 2024-09-13T12:16:18.245 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 4196072543 2024-09-13T12:16:18.245 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.245 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:16:18.245 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.245 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addr": "127.0.0.1:6803/4196072543", 2024-09-13T12:16:18.245 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addr": "127.0.0.1:6805/4196072543", 2024-09-13T12:16:18.245 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addr": "127.0.0.1:6809/4196072543", 2024-09-13T12:16:18.245 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addr": "127.0.0.1:6807/4196072543", 2024-09-13T12:16:18.245 INFO:tasks.workunit.client.0.smithi032.stdout: "state": [ 2024-09-13T12:16:18.245 INFO:tasks.workunit.client.0.smithi032.stdout: "exists", 2024-09-13T12:16:18.245 INFO:tasks.workunit.client.0.smithi032.stdout: "up" 2024-09-13T12:16:18.245 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:16:18.245 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.246 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.246 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 1, 2024-09-13T12:16:18.246 INFO:tasks.workunit.client.0.smithi032.stdout: "uuid": "b6fb1428-41f7-46eb-8099-15ef8f53355b", 2024-09-13T12:16:18.246 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 0, 2024-09-13T12:16:18.246 INFO:tasks.workunit.client.0.smithi032.stdout: "in": 1, 2024-09-13T12:16:18.246 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 1, 2024-09-13T12:16:18.246 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_affinity": 1, 2024-09-13T12:16:18.246 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_begin": 0, 2024-09-13T12:16:18.246 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_end": 0, 2024-09-13T12:16:18.246 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 10, 2024-09-13T12:16:18.246 INFO:tasks.workunit.client.0.smithi032.stdout: "up_thru": 25, 2024-09-13T12:16:18.246 INFO:tasks.workunit.client.0.smithi032.stdout: "down_at": 32, 2024-09-13T12:16:18.246 INFO:tasks.workunit.client.0.smithi032.stdout: "lost_at": 0, 2024-09-13T12:16:18.246 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addrs": { 2024-09-13T12:16:18.246 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:16:18.247 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.247 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:16:18.247 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6810", 2024-09-13T12:16:18.247 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3755373257 2024-09-13T12:16:18.247 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.247 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.247 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:16:18.247 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6811", 2024-09-13T12:16:18.247 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3755373257 2024-09-13T12:16:18.247 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.247 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:16:18.247 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.247 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addrs": { 2024-09-13T12:16:18.247 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:16:18.247 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.248 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:16:18.248 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6812", 2024-09-13T12:16:18.248 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3755373257 2024-09-13T12:16:18.248 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.248 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.248 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:16:18.248 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6813", 2024-09-13T12:16:18.248 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3755373257 2024-09-13T12:16:18.248 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.248 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:16:18.248 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.248 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-13T12:16:18.248 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:16:18.249 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.249 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:16:18.249 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6816", 2024-09-13T12:16:18.249 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3755373257 2024-09-13T12:16:18.249 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.249 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.249 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:16:18.249 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6817", 2024-09-13T12:16:18.249 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3755373257 2024-09-13T12:16:18.249 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.249 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:16:18.249 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.249 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-13T12:16:18.249 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:16:18.250 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.250 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:16:18.250 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6814", 2024-09-13T12:16:18.250 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3755373257 2024-09-13T12:16:18.250 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.250 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.250 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:16:18.250 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6815", 2024-09-13T12:16:18.250 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3755373257 2024-09-13T12:16:18.250 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.250 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:16:18.250 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.250 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addr": "127.0.0.1:6811/3755373257", 2024-09-13T12:16:18.250 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addr": "127.0.0.1:6813/3755373257", 2024-09-13T12:16:18.251 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3755373257", 2024-09-13T12:16:18.251 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3755373257", 2024-09-13T12:16:18.251 INFO:tasks.workunit.client.0.smithi032.stdout: "state": [ 2024-09-13T12:16:18.251 INFO:tasks.workunit.client.0.smithi032.stdout: "exists" 2024-09-13T12:16:18.251 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:16:18.251 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.251 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.251 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2, 2024-09-13T12:16:18.251 INFO:tasks.workunit.client.0.smithi032.stdout: "uuid": "575c91a2-dd41-4141-bbec-f3711b8a1a2e", 2024-09-13T12:16:18.251 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 1, 2024-09-13T12:16:18.251 INFO:tasks.workunit.client.0.smithi032.stdout: "in": 1, 2024-09-13T12:16:18.251 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 1, 2024-09-13T12:16:18.251 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_affinity": 1, 2024-09-13T12:16:18.251 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_begin": 15, 2024-09-13T12:16:18.252 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_end": 29, 2024-09-13T12:16:18.252 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 39, 2024-09-13T12:16:18.252 INFO:tasks.workunit.client.0.smithi032.stdout: "up_thru": 0, 2024-09-13T12:16:18.252 INFO:tasks.workunit.client.0.smithi032.stdout: "down_at": 33, 2024-09-13T12:16:18.252 INFO:tasks.workunit.client.0.smithi032.stdout: "lost_at": 0, 2024-09-13T12:16:18.252 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addrs": { 2024-09-13T12:16:18.252 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:16:18.252 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.252 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:16:18.252 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6810", 2024-09-13T12:16:18.252 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3815047400 2024-09-13T12:16:18.252 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.252 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.252 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:16:18.252 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6811", 2024-09-13T12:16:18.253 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3815047400 2024-09-13T12:16:18.253 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.253 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:16:18.253 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.253 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addrs": { 2024-09-13T12:16:18.253 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:16:18.253 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.253 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:16:18.253 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6812", 2024-09-13T12:16:18.253 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3815047400 2024-09-13T12:16:18.253 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.253 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.253 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:16:18.253 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6813", 2024-09-13T12:16:18.254 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3815047400 2024-09-13T12:16:18.254 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.254 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:16:18.254 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.254 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-13T12:16:18.254 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:16:18.254 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.254 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:16:18.254 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6816", 2024-09-13T12:16:18.254 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3815047400 2024-09-13T12:16:18.254 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.254 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.254 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:16:18.254 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6817", 2024-09-13T12:16:18.254 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3815047400 2024-09-13T12:16:18.255 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.255 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:16:18.255 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.255 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-13T12:16:18.255 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:16:18.255 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.255 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:16:18.255 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6814", 2024-09-13T12:16:18.255 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3815047400 2024-09-13T12:16:18.255 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.255 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.255 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:16:18.255 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6815", 2024-09-13T12:16:18.255 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3815047400 2024-09-13T12:16:18.256 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.256 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:16:18.256 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.256 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addr": "127.0.0.1:6811/3815047400", 2024-09-13T12:16:18.256 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addr": "127.0.0.1:6813/3815047400", 2024-09-13T12:16:18.256 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3815047400", 2024-09-13T12:16:18.256 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3815047400", 2024-09-13T12:16:18.256 INFO:tasks.workunit.client.0.smithi032.stdout: "state": [ 2024-09-13T12:16:18.256 INFO:tasks.workunit.client.0.smithi032.stdout: "exists", 2024-09-13T12:16:18.256 INFO:tasks.workunit.client.0.smithi032.stdout: "up" 2024-09-13T12:16:18.256 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:16:18.256 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.256 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:16:18.257 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_xinfo": [ 2024-09-13T12:16:18.257 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.257 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 0, 2024-09-13T12:16:18.257 INFO:tasks.workunit.client.0.smithi032.stdout: "down_stamp": "2024-09-13T12:15:20.684314+0000", 2024-09-13T12:16:18.257 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_probability": 0, 2024-09-13T12:16:18.257 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_interval": 0, 2024-09-13T12:16:18.257 INFO:tasks.workunit.client.0.smithi032.stdout: "features": 4540719139924082687, 2024-09-13T12:16:18.257 INFO:tasks.workunit.client.0.smithi032.stdout: "old_weight": 0, 2024-09-13T12:16:18.257 INFO:tasks.workunit.client.0.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-13T12:13:04.824763+0000", 2024-09-13T12:16:18.257 INFO:tasks.workunit.client.0.smithi032.stdout: "dead_epoch": 0 2024-09-13T12:16:18.257 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.257 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.257 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 1, 2024-09-13T12:16:18.257 INFO:tasks.workunit.client.0.smithi032.stdout: "down_stamp": "2024-09-13T12:15:21.265408+0000", 2024-09-13T12:16:18.258 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_probability": 0, 2024-09-13T12:16:18.258 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_interval": 0, 2024-09-13T12:16:18.258 INFO:tasks.workunit.client.0.smithi032.stdout: "features": 4540719139924082687, 2024-09-13T12:16:18.258 INFO:tasks.workunit.client.0.smithi032.stdout: "old_weight": 0, 2024-09-13T12:16:18.258 INFO:tasks.workunit.client.0.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-13T12:13:41.500292+0000", 2024-09-13T12:16:18.258 INFO:tasks.workunit.client.0.smithi032.stdout: "dead_epoch": 0 2024-09-13T12:16:18.258 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.258 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.258 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2, 2024-09-13T12:16:18.258 INFO:tasks.workunit.client.0.smithi032.stdout: "down_stamp": "2024-09-13T12:15:21.791674+0000", 2024-09-13T12:16:18.258 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_probability": 0, 2024-09-13T12:16:18.258 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_interval": 0, 2024-09-13T12:16:18.258 INFO:tasks.workunit.client.0.smithi032.stdout: "features": 4540719139924082687, 2024-09-13T12:16:18.258 INFO:tasks.workunit.client.0.smithi032.stdout: "old_weight": 0, 2024-09-13T12:16:18.259 INFO:tasks.workunit.client.0.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-13T12:14:17.639955+0000", 2024-09-13T12:16:18.259 INFO:tasks.workunit.client.0.smithi032.stdout: "dead_epoch": 0 2024-09-13T12:16:18.259 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.259 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:16:18.259 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_upmap": [], 2024-09-13T12:16:18.259 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_upmap_items": [], 2024-09-13T12:16:18.259 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_upmap_primaries": [], 2024-09-13T12:16:18.259 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_temp": [], 2024-09-13T12:16:18.259 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_temp": [], 2024-09-13T12:16:18.259 INFO:tasks.workunit.client.0.smithi032.stdout: "blocklist": {}, 2024-09-13T12:16:18.259 INFO:tasks.workunit.client.0.smithi032.stdout: "range_blocklist": {}, 2024-09-13T12:16:18.259 INFO:tasks.workunit.client.0.smithi032.stdout: "erasure_code_profiles": { 2024-09-13T12:16:18.259 INFO:tasks.workunit.client.0.smithi032.stdout: "default": { 2024-09-13T12:16:18.259 INFO:tasks.workunit.client.0.smithi032.stdout: "k": "2", 2024-09-13T12:16:18.259 INFO:tasks.workunit.client.0.smithi032.stdout: "m": "2", 2024-09-13T12:16:18.260 INFO:tasks.workunit.client.0.smithi032.stdout: "plugin": "jerasure", 2024-09-13T12:16:18.260 INFO:tasks.workunit.client.0.smithi032.stdout: "technique": "reed_sol_van" 2024-09-13T12:16:18.260 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.260 INFO:tasks.workunit.client.0.smithi032.stdout: "myprofile": { 2024-09-13T12:16:18.260 INFO:tasks.workunit.client.0.smithi032.stdout: "crush-device-class": "", 2024-09-13T12:16:18.260 INFO:tasks.workunit.client.0.smithi032.stdout: "crush-failure-domain": "osd", 2024-09-13T12:16:18.260 INFO:tasks.workunit.client.0.smithi032.stdout: "crush-num-failure-domains": "0", 2024-09-13T12:16:18.260 INFO:tasks.workunit.client.0.smithi032.stdout: "crush-osds-per-failure-domain": "0", 2024-09-13T12:16:18.260 INFO:tasks.workunit.client.0.smithi032.stdout: "crush-root": "default", 2024-09-13T12:16:18.260 INFO:tasks.workunit.client.0.smithi032.stdout: "jerasure-per-chunk-alignment": "false", 2024-09-13T12:16:18.260 INFO:tasks.workunit.client.0.smithi032.stdout: "k": "2", 2024-09-13T12:16:18.260 INFO:tasks.workunit.client.0.smithi032.stdout: "m": "1", 2024-09-13T12:16:18.260 INFO:tasks.workunit.client.0.smithi032.stdout: "plugin": "jerasure", 2024-09-13T12:16:18.260 INFO:tasks.workunit.client.0.smithi032.stdout: "technique": "reed_sol_van", 2024-09-13T12:16:18.261 INFO:tasks.workunit.client.0.smithi032.stdout: "w": "8" 2024-09-13T12:16:18.261 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.261 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.261 INFO:tasks.workunit.client.0.smithi032.stdout: "removed_snaps_queue": [], 2024-09-13T12:16:18.261 INFO:tasks.workunit.client.0.smithi032.stdout: "new_removed_snaps": [], 2024-09-13T12:16:18.261 INFO:tasks.workunit.client.0.smithi032.stdout: "new_purged_snaps": [], 2024-09-13T12:16:18.261 INFO:tasks.workunit.client.0.smithi032.stdout: "crush_node_flags": {}, 2024-09-13T12:16:18.261 INFO:tasks.workunit.client.0.smithi032.stdout: "device_class_flags": {}, 2024-09-13T12:16:18.261 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_mode": { 2024-09-13T12:16:18.261 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_mode_enabled": false, 2024-09-13T12:16:18.261 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_bucket_count": 0, 2024-09-13T12:16:18.261 INFO:tasks.workunit.client.0.smithi032.stdout: "degraded_stretch_mode": 0, 2024-09-13T12:16:18.261 INFO:tasks.workunit.client.0.smithi032.stdout: "recovering_stretch_mode": 0, 2024-09-13T12:16:18.261 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_mode_bucket": 0 2024-09-13T12:16:18.262 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.262 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.262 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_metadata": [ 2024-09-13T12:16:18.262 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.262 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:16:18.262 INFO:tasks.workunit.client.0.smithi032.stdout: "arch": "x86_64", 2024-09-13T12:16:18.262 INFO:tasks.workunit.client.0.smithi032.stdout: "back_addr": "[v2:127.0.0.1:6804/4196072543,v1:127.0.0.1:6805/4196072543]", 2024-09-13T12:16:18.262 INFO:tasks.workunit.client.0.smithi032.stdout: "back_iface": "", 2024-09-13T12:16:18.262 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs": "1", 2024-09-13T12:16:18.262 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_db": "0", 2024-09-13T12:16:18.262 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_wal": "0", 2024-09-13T12:16:18.262 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_single_shared_device": "1", 2024-09-13T12:16:18.262 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_allocation_from_file": "0", 2024-09-13T12:16:18.262 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_access_mode": "file", 2024-09-13T12:16:18.263 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_block_size": "4096", 2024-09-13T12:16:18.263 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_devices": "sda", 2024-09-13T12:16:18.263 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-13T12:16:18.263 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-13T12:16:18.263 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-13T12:16:18.263 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_rotational": "1", 2024-09-13T12:16:18.263 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_size": "107374182400", 2024-09-13T12:16:18.263 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_support_discard": "1", 2024-09-13T12:16:18.263 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_type": "hdd", 2024-09-13T12:16:18.263 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_min_alloc_size": "4096", 2024-09-13T12:16:18.263 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_release": "squid", 2024-09-13T12:16:18.263 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:16:18.263 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_short": "19.3.0-4697-gd58f7306", 2024-09-13T12:16:18.263 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_when_created": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:16:18.263 INFO:tasks.workunit.client.0.smithi032.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-13T12:16:18.264 INFO:tasks.workunit.client.0.smithi032.stdout: "created_at": "2024-09-13T12:12:37.432453Z", 2024-09-13T12:16:18.264 INFO:tasks.workunit.client.0.smithi032.stdout: "default_device_class": "hdd", 2024-09-13T12:16:18.264 INFO:tasks.workunit.client.0.smithi032.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4J2LL", 2024-09-13T12:16:18.264 INFO:tasks.workunit.client.0.smithi032.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-13T12:16:18.264 INFO:tasks.workunit.client.0.smithi032.stdout: "devices": "sda", 2024-09-13T12:16:18.264 INFO:tasks.workunit.client.0.smithi032.stdout: "distro": "centos", 2024-09-13T12:16:18.264 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_description": "CentOS Stream 9", 2024-09-13T12:16:18.264 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_version": "9", 2024-09-13T12:16:18.264 INFO:tasks.workunit.client.0.smithi032.stdout: "front_addr": "[v2:127.0.0.1:6802/4196072543,v1:127.0.0.1:6803/4196072543]", 2024-09-13T12:16:18.264 INFO:tasks.workunit.client.0.smithi032.stdout: "front_iface": "", 2024-09-13T12:16:18.264 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/4196072543,v1:127.0.0.1:6809/4196072543]", 2024-09-13T12:16:18.264 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/4196072543,v1:127.0.0.1:6807/4196072543]", 2024-09-13T12:16:18.264 INFO:tasks.workunit.client.0.smithi032.stdout: "hostname": "smithi032", 2024-09-13T12:16:18.265 INFO:tasks.workunit.client.0.smithi032.stdout: "journal_rotational": "1", 2024-09-13T12:16:18.265 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-13T12:16:18.265 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-13T12:16:18.265 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_swap_kb": "0", 2024-09-13T12:16:18.265 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_total_kb": "32493484", 2024-09-13T12:16:18.265 INFO:tasks.workunit.client.0.smithi032.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-13T12:16:18.265 INFO:tasks.workunit.client.0.smithi032.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-13T12:16:18.265 INFO:tasks.workunit.client.0.smithi032.stdout: "os": "Linux", 2024-09-13T12:16:18.265 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_data": "td/divergent-priors/0", 2024-09-13T12:16:18.265 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_objectstore": "bluestore", 2024-09-13T12:16:18.265 INFO:tasks.workunit.client.0.smithi032.stdout: "osdspec_affinity": "", 2024-09-13T12:16:18.265 INFO:tasks.workunit.client.0.smithi032.stdout: "rotational": "1" 2024-09-13T12:16:18.265 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.265 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.266 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 1, 2024-09-13T12:16:18.266 INFO:tasks.workunit.client.0.smithi032.stdout: "arch": "x86_64", 2024-09-13T12:16:18.266 INFO:tasks.workunit.client.0.smithi032.stdout: "back_addr": "[v2:127.0.0.1:6812/3755373257,v1:127.0.0.1:6813/3755373257]", 2024-09-13T12:16:18.266 INFO:tasks.workunit.client.0.smithi032.stdout: "back_iface": "", 2024-09-13T12:16:18.266 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs": "1", 2024-09-13T12:16:18.266 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_db": "0", 2024-09-13T12:16:18.266 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_wal": "0", 2024-09-13T12:16:18.266 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_single_shared_device": "1", 2024-09-13T12:16:18.266 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_allocation_from_file": "0", 2024-09-13T12:16:18.266 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_access_mode": "file", 2024-09-13T12:16:18.266 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_block_size": "4096", 2024-09-13T12:16:18.266 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_devices": "sda", 2024-09-13T12:16:18.266 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-13T12:16:18.267 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-13T12:16:18.267 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-13T12:16:18.267 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_rotational": "1", 2024-09-13T12:16:18.267 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_size": "107374182400", 2024-09-13T12:16:18.267 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_support_discard": "1", 2024-09-13T12:16:18.267 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_type": "hdd", 2024-09-13T12:16:18.267 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_min_alloc_size": "4096", 2024-09-13T12:16:18.267 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_release": "squid", 2024-09-13T12:16:18.267 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:16:18.267 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_short": "19.3.0-4697-gd58f7306", 2024-09-13T12:16:18.267 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_when_created": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:16:18.267 INFO:tasks.workunit.client.0.smithi032.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-13T12:16:18.267 INFO:tasks.workunit.client.0.smithi032.stdout: "created_at": "2024-09-13T12:13:12.366247Z", 2024-09-13T12:16:18.267 INFO:tasks.workunit.client.0.smithi032.stdout: "default_device_class": "hdd", 2024-09-13T12:16:18.268 INFO:tasks.workunit.client.0.smithi032.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4J2LL", 2024-09-13T12:16:18.268 INFO:tasks.workunit.client.0.smithi032.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-13T12:16:18.268 INFO:tasks.workunit.client.0.smithi032.stdout: "devices": "sda", 2024-09-13T12:16:18.268 INFO:tasks.workunit.client.0.smithi032.stdout: "distro": "centos", 2024-09-13T12:16:18.268 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_description": "CentOS Stream 9", 2024-09-13T12:16:18.268 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_version": "9", 2024-09-13T12:16:18.268 INFO:tasks.workunit.client.0.smithi032.stdout: "front_addr": "[v2:127.0.0.1:6810/3755373257,v1:127.0.0.1:6811/3755373257]", 2024-09-13T12:16:18.268 INFO:tasks.workunit.client.0.smithi032.stdout: "front_iface": "", 2024-09-13T12:16:18.268 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3755373257,v1:127.0.0.1:6817/3755373257]", 2024-09-13T12:16:18.268 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3755373257,v1:127.0.0.1:6815/3755373257]", 2024-09-13T12:16:18.268 INFO:tasks.workunit.client.0.smithi032.stdout: "hostname": "smithi032", 2024-09-13T12:16:18.268 INFO:tasks.workunit.client.0.smithi032.stdout: "journal_rotational": "1", 2024-09-13T12:16:18.268 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-13T12:16:18.268 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-13T12:16:18.268 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_swap_kb": "0", 2024-09-13T12:16:18.269 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_total_kb": "32493484", 2024-09-13T12:16:18.269 INFO:tasks.workunit.client.0.smithi032.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-13T12:16:18.269 INFO:tasks.workunit.client.0.smithi032.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-13T12:16:18.269 INFO:tasks.workunit.client.0.smithi032.stdout: "os": "Linux", 2024-09-13T12:16:18.269 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_data": "td/divergent-priors/1", 2024-09-13T12:16:18.269 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_objectstore": "bluestore", 2024-09-13T12:16:18.269 INFO:tasks.workunit.client.0.smithi032.stdout: "osdspec_affinity": "", 2024-09-13T12:16:18.269 INFO:tasks.workunit.client.0.smithi032.stdout: "rotational": "1" 2024-09-13T12:16:18.269 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.269 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.269 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:16:18.269 INFO:tasks.workunit.client.0.smithi032.stdout: "arch": "x86_64", 2024-09-13T12:16:18.269 INFO:tasks.workunit.client.0.smithi032.stdout: "back_addr": "[v2:127.0.0.1:6812/3815047400,v1:127.0.0.1:6813/3815047400]", 2024-09-13T12:16:18.269 INFO:tasks.workunit.client.0.smithi032.stdout: "back_iface": "", 2024-09-13T12:16:18.270 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs": "1", 2024-09-13T12:16:18.270 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_db": "0", 2024-09-13T12:16:18.270 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_wal": "0", 2024-09-13T12:16:18.270 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_single_shared_device": "1", 2024-09-13T12:16:18.270 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_allocation_from_file": "0", 2024-09-13T12:16:18.270 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_access_mode": "file", 2024-09-13T12:16:18.270 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_block_size": "4096", 2024-09-13T12:16:18.270 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_devices": "sda", 2024-09-13T12:16:18.270 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-13T12:16:18.270 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-13T12:16:18.270 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-13T12:16:18.270 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_rotational": "1", 2024-09-13T12:16:18.270 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_size": "107374182400", 2024-09-13T12:16:18.270 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_support_discard": "1", 2024-09-13T12:16:18.271 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_type": "hdd", 2024-09-13T12:16:18.271 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_min_alloc_size": "4096", 2024-09-13T12:16:18.271 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_release": "squid", 2024-09-13T12:16:18.271 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:16:18.271 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_short": "19.3.0-4697-gd58f7306", 2024-09-13T12:16:18.271 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_when_created": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:16:18.271 INFO:tasks.workunit.client.0.smithi032.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-13T12:16:18.271 INFO:tasks.workunit.client.0.smithi032.stdout: "created_at": "2024-09-13T12:13:49.134953Z", 2024-09-13T12:16:18.271 INFO:tasks.workunit.client.0.smithi032.stdout: "default_device_class": "hdd", 2024-09-13T12:16:18.271 INFO:tasks.workunit.client.0.smithi032.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4J2LL", 2024-09-13T12:16:18.271 INFO:tasks.workunit.client.0.smithi032.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-13T12:16:18.271 INFO:tasks.workunit.client.0.smithi032.stdout: "devices": "sda", 2024-09-13T12:16:18.271 INFO:tasks.workunit.client.0.smithi032.stdout: "distro": "centos", 2024-09-13T12:16:18.271 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_description": "CentOS Stream 9", 2024-09-13T12:16:18.272 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_version": "9", 2024-09-13T12:16:18.272 INFO:tasks.workunit.client.0.smithi032.stdout: "front_addr": "[v2:127.0.0.1:6810/3815047400,v1:127.0.0.1:6811/3815047400]", 2024-09-13T12:16:18.272 INFO:tasks.workunit.client.0.smithi032.stdout: "front_iface": "", 2024-09-13T12:16:18.272 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3815047400,v1:127.0.0.1:6817/3815047400]", 2024-09-13T12:16:18.272 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3815047400,v1:127.0.0.1:6815/3815047400]", 2024-09-13T12:16:18.272 INFO:tasks.workunit.client.0.smithi032.stdout: "hostname": "smithi032", 2024-09-13T12:16:18.272 INFO:tasks.workunit.client.0.smithi032.stdout: "journal_rotational": "1", 2024-09-13T12:16:18.272 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-13T12:16:18.272 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-13T12:16:18.272 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_swap_kb": "0", 2024-09-13T12:16:18.272 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_total_kb": "32493484", 2024-09-13T12:16:18.272 INFO:tasks.workunit.client.0.smithi032.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-13T12:16:18.272 INFO:tasks.workunit.client.0.smithi032.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-13T12:16:18.273 INFO:tasks.workunit.client.0.smithi032.stdout: "os": "Linux", 2024-09-13T12:16:18.273 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_data": "td/divergent-priors/2", 2024-09-13T12:16:18.273 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_objectstore": "bluestore", 2024-09-13T12:16:18.273 INFO:tasks.workunit.client.0.smithi032.stdout: "osdspec_affinity": "", 2024-09-13T12:16:18.273 INFO:tasks.workunit.client.0.smithi032.stdout: "rotational": "1" 2024-09-13T12:16:18.273 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.273 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:16:18.273 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap_clean_epochs": { 2024-09-13T12:16:18.273 INFO:tasks.workunit.client.0.smithi032.stdout: "min_last_epoch_clean": 0, 2024-09-13T12:16:18.273 INFO:tasks.workunit.client.0.smithi032.stdout: "last_epoch_clean": { 2024-09-13T12:16:18.273 INFO:tasks.workunit.client.0.smithi032.stdout: "per_pool": [] 2024-09-13T12:16:18.273 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.273 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_epochs": [ 2024-09-13T12:16:18.273 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.273 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:16:18.274 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 37 2024-09-13T12:16:18.274 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.274 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.274 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:16:18.274 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 40 2024-09-13T12:16:18.274 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.274 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:16:18.274 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.274 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap_first_committed": 1, 2024-09-13T12:16:18.274 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap_last_committed": 42, 2024-09-13T12:16:18.274 INFO:tasks.workunit.client.0.smithi032.stdout: "crushmap": { 2024-09-13T12:16:18.274 INFO:tasks.workunit.client.0.smithi032.stdout: "devices": [ 2024-09-13T12:16:18.274 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.275 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:16:18.275 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "osd.0", 2024-09-13T12:16:18.275 INFO:tasks.workunit.client.0.smithi032.stdout: "class": "hdd" 2024-09-13T12:16:18.275 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.275 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.275 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 1, 2024-09-13T12:16:18.275 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "osd.1", 2024-09-13T12:16:18.275 INFO:tasks.workunit.client.0.smithi032.stdout: "class": "hdd" 2024-09-13T12:16:18.275 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.275 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.275 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:16:18.275 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "osd.2", 2024-09-13T12:16:18.275 INFO:tasks.workunit.client.0.smithi032.stdout: "class": "hdd" 2024-09-13T12:16:18.275 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.276 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:16:18.276 INFO:tasks.workunit.client.0.smithi032.stdout: "types": [ 2024-09-13T12:16:18.276 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.276 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 0, 2024-09-13T12:16:18.276 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "osd" 2024-09-13T12:16:18.276 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.276 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.276 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 1, 2024-09-13T12:16:18.276 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "host" 2024-09-13T12:16:18.276 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.276 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.276 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 2, 2024-09-13T12:16:18.276 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "chassis" 2024-09-13T12:16:18.277 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.277 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.277 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 3, 2024-09-13T12:16:18.277 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "rack" 2024-09-13T12:16:18.277 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.277 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.277 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 4, 2024-09-13T12:16:18.277 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "row" 2024-09-13T12:16:18.277 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.277 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.277 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 5, 2024-09-13T12:16:18.277 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "pdu" 2024-09-13T12:16:18.277 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.277 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.278 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 6, 2024-09-13T12:16:18.278 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "pod" 2024-09-13T12:16:18.278 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.278 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.278 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 7, 2024-09-13T12:16:18.278 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "room" 2024-09-13T12:16:18.278 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.278 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.278 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 8, 2024-09-13T12:16:18.278 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "datacenter" 2024-09-13T12:16:18.278 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.278 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.278 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 9, 2024-09-13T12:16:18.278 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "zone" 2024-09-13T12:16:18.279 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.279 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.279 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 10, 2024-09-13T12:16:18.279 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "region" 2024-09-13T12:16:18.279 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.279 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.279 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 11, 2024-09-13T12:16:18.279 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "root" 2024-09-13T12:16:18.279 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.279 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:16:18.279 INFO:tasks.workunit.client.0.smithi032.stdout: "buckets": [ 2024-09-13T12:16:18.279 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.279 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -1, 2024-09-13T12:16:18.279 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "default", 2024-09-13T12:16:18.280 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 11, 2024-09-13T12:16:18.280 INFO:tasks.workunit.client.0.smithi032.stdout: "type_name": "root", 2024-09-13T12:16:18.280 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:16:18.280 INFO:tasks.workunit.client.0.smithi032.stdout: "alg": "straw2", 2024-09-13T12:16:18.280 INFO:tasks.workunit.client.0.smithi032.stdout: "hash": "rjenkins1", 2024-09-13T12:16:18.280 INFO:tasks.workunit.client.0.smithi032.stdout: "items": [ 2024-09-13T12:16:18.280 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.280 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -3, 2024-09-13T12:16:18.280 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:16:18.280 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 0 2024-09-13T12:16:18.280 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.280 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:16:18.280 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.281 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.281 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -2, 2024-09-13T12:16:18.281 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "default~hdd", 2024-09-13T12:16:18.281 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 11, 2024-09-13T12:16:18.281 INFO:tasks.workunit.client.0.smithi032.stdout: "type_name": "root", 2024-09-13T12:16:18.281 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:16:18.281 INFO:tasks.workunit.client.0.smithi032.stdout: "alg": "straw2", 2024-09-13T12:16:18.281 INFO:tasks.workunit.client.0.smithi032.stdout: "hash": "rjenkins1", 2024-09-13T12:16:18.281 INFO:tasks.workunit.client.0.smithi032.stdout: "items": [ 2024-09-13T12:16:18.281 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.281 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -4, 2024-09-13T12:16:18.281 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:16:18.281 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 0 2024-09-13T12:16:18.281 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.281 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:16:18.282 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.282 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.282 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -3, 2024-09-13T12:16:18.282 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "smithi032", 2024-09-13T12:16:18.282 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 1, 2024-09-13T12:16:18.284 INFO:tasks.workunit.client.0.smithi032.stdout: "type_name": "host", 2024-09-13T12:16:18.284 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:16:18.284 INFO:tasks.workunit.client.0.smithi032.stdout: "alg": "straw2", 2024-09-13T12:16:18.284 INFO:tasks.workunit.client.0.smithi032.stdout: "hash": "rjenkins1", 2024-09-13T12:16:18.284 INFO:tasks.workunit.client.0.smithi032.stdout: "items": [ 2024-09-13T12:16:18.284 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.284 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:16:18.284 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:16:18.285 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 0 2024-09-13T12:16:18.285 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.285 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.285 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 1, 2024-09-13T12:16:18.285 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:16:18.285 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 1 2024-09-13T12:16:18.285 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.285 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.285 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:16:18.285 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:16:18.285 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 2 2024-09-13T12:16:18.285 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.285 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:16:18.285 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.286 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.286 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -4, 2024-09-13T12:16:18.286 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "smithi032~hdd", 2024-09-13T12:16:18.286 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 1, 2024-09-13T12:16:18.286 INFO:tasks.workunit.client.0.smithi032.stdout: "type_name": "host", 2024-09-13T12:16:18.286 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:16:18.286 INFO:tasks.workunit.client.0.smithi032.stdout: "alg": "straw2", 2024-09-13T12:16:18.286 INFO:tasks.workunit.client.0.smithi032.stdout: "hash": "rjenkins1", 2024-09-13T12:16:18.286 INFO:tasks.workunit.client.0.smithi032.stdout: "items": [ 2024-09-13T12:16:18.286 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.286 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:16:18.286 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:16:18.286 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 0 2024-09-13T12:16:18.286 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.286 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.287 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 1, 2024-09-13T12:16:18.287 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:16:18.287 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 1 2024-09-13T12:16:18.287 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.287 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.287 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:16:18.287 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:16:18.287 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 2 2024-09-13T12:16:18.287 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.287 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:16:18.287 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.287 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:16:18.287 INFO:tasks.workunit.client.0.smithi032.stdout: "rules": [ 2024-09-13T12:16:18.287 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.287 INFO:tasks.workunit.client.0.smithi032.stdout: "rule_id": 0, 2024-09-13T12:16:18.288 INFO:tasks.workunit.client.0.smithi032.stdout: "rule_name": "replicated_rule", 2024-09-13T12:16:18.288 INFO:tasks.workunit.client.0.smithi032.stdout: "type": 1, 2024-09-13T12:16:18.288 INFO:tasks.workunit.client.0.smithi032.stdout: "steps": [ 2024-09-13T12:16:18.288 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.288 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "take", 2024-09-13T12:16:18.288 INFO:tasks.workunit.client.0.smithi032.stdout: "item": -1, 2024-09-13T12:16:18.288 INFO:tasks.workunit.client.0.smithi032.stdout: "item_name": "default" 2024-09-13T12:16:18.288 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.288 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.288 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "choose_firstn", 2024-09-13T12:16:18.288 INFO:tasks.workunit.client.0.smithi032.stdout: "num": 0, 2024-09-13T12:16:18.288 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "osd" 2024-09-13T12:16:18.288 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.288 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.289 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "emit" 2024-09-13T12:16:18.289 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.289 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:16:18.289 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.289 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.289 INFO:tasks.workunit.client.0.smithi032.stdout: "rule_id": 1, 2024-09-13T12:16:18.289 INFO:tasks.workunit.client.0.smithi032.stdout: "rule_name": "test", 2024-09-13T12:16:18.289 INFO:tasks.workunit.client.0.smithi032.stdout: "type": 3, 2024-09-13T12:16:18.289 INFO:tasks.workunit.client.0.smithi032.stdout: "steps": [ 2024-09-13T12:16:18.289 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.289 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "set_chooseleaf_tries", 2024-09-13T12:16:18.289 INFO:tasks.workunit.client.0.smithi032.stdout: "num": 5 2024-09-13T12:16:18.289 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.289 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.289 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "set_choose_tries", 2024-09-13T12:16:18.290 INFO:tasks.workunit.client.0.smithi032.stdout: "num": 100 2024-09-13T12:16:18.290 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.290 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.290 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "take", 2024-09-13T12:16:18.290 INFO:tasks.workunit.client.0.smithi032.stdout: "item": -1, 2024-09-13T12:16:18.290 INFO:tasks.workunit.client.0.smithi032.stdout: "item_name": "default" 2024-09-13T12:16:18.290 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.290 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.290 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "choose_indep", 2024-09-13T12:16:18.290 INFO:tasks.workunit.client.0.smithi032.stdout: "num": 0, 2024-09-13T12:16:18.290 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "osd" 2024-09-13T12:16:18.290 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.290 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.290 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "emit" 2024-09-13T12:16:18.291 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.291 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:16:18.291 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.291 INFO:tasks.workunit.client.0.smithi032.stderr:report 730243173 2024-09-13T12:16:18.291 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:16:18.292 INFO:tasks.workunit.client.0.smithi032.stdout: "tunables": { 2024-09-13T12:16:18.292 INFO:tasks.workunit.client.0.smithi032.stdout: "choose_local_tries": 0, 2024-09-13T12:16:18.292 INFO:tasks.workunit.client.0.smithi032.stdout: "choose_local_fallback_tries": 0, 2024-09-13T12:16:18.292 INFO:tasks.workunit.client.0.smithi032.stdout: "choose_total_tries": 50, 2024-09-13T12:16:18.292 INFO:tasks.workunit.client.0.smithi032.stdout: "chooseleaf_descend_once": 1, 2024-09-13T12:16:18.292 INFO:tasks.workunit.client.0.smithi032.stdout: "chooseleaf_vary_r": 1, 2024-09-13T12:16:18.292 INFO:tasks.workunit.client.0.smithi032.stdout: "chooseleaf_stable": 1, 2024-09-13T12:16:18.292 INFO:tasks.workunit.client.0.smithi032.stdout: "msr_descents": 100, 2024-09-13T12:16:18.292 INFO:tasks.workunit.client.0.smithi032.stdout: "msr_collision_tries": 100, 2024-09-13T12:16:18.292 INFO:tasks.workunit.client.0.smithi032.stdout: "straw_calc_version": 1, 2024-09-13T12:16:18.292 INFO:tasks.workunit.client.0.smithi032.stdout: "allowed_bucket_algs": 54, 2024-09-13T12:16:18.292 INFO:tasks.workunit.client.0.smithi032.stdout: "profile": "jewel", 2024-09-13T12:16:18.292 INFO:tasks.workunit.client.0.smithi032.stdout: "optimal_tunables": 1, 2024-09-13T12:16:18.292 INFO:tasks.workunit.client.0.smithi032.stdout: "legacy_tunables": 0, 2024-09-13T12:16:18.293 INFO:tasks.workunit.client.0.smithi032.stdout: "minimum_required_version": "jewel", 2024-09-13T12:16:18.293 INFO:tasks.workunit.client.0.smithi032.stdout: "require_feature_tunables": 1, 2024-09-13T12:16:18.293 INFO:tasks.workunit.client.0.smithi032.stdout: "require_feature_tunables2": 1, 2024-09-13T12:16:18.293 INFO:tasks.workunit.client.0.smithi032.stdout: "has_v2_rules": 1, 2024-09-13T12:16:18.293 INFO:tasks.workunit.client.0.smithi032.stdout: "require_feature_tunables3": 1, 2024-09-13T12:16:18.293 INFO:tasks.workunit.client.0.smithi032.stdout: "has_v3_rules": 0, 2024-09-13T12:16:18.293 INFO:tasks.workunit.client.0.smithi032.stdout: "has_v4_buckets": 1, 2024-09-13T12:16:18.293 INFO:tasks.workunit.client.0.smithi032.stdout: "require_feature_tunables5": 1, 2024-09-13T12:16:18.293 INFO:tasks.workunit.client.0.smithi032.stdout: "has_v5_rules": 0, 2024-09-13T12:16:18.293 INFO:tasks.workunit.client.0.smithi032.stdout: "has_msr_rules": 0 2024-09-13T12:16:18.293 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.293 INFO:tasks.workunit.client.0.smithi032.stdout: "choose_args": {} 2024-09-13T12:16:18.293 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.293 INFO:tasks.workunit.client.0.smithi032.stdout: "fsmap": { 2024-09-13T12:16:18.294 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 1, 2024-09-13T12:16:18.294 INFO:tasks.workunit.client.0.smithi032.stdout: "btime": "2024-09-13T12:12:31:260139+0000", 2024-09-13T12:16:18.294 INFO:tasks.workunit.client.0.smithi032.stdout: "default_fscid": -1, 2024-09-13T12:16:18.294 INFO:tasks.workunit.client.0.smithi032.stdout: "compat": { 2024-09-13T12:16:18.294 INFO:tasks.workunit.client.0.smithi032.stdout: "compat": {}, 2024-09-13T12:16:18.294 INFO:tasks.workunit.client.0.smithi032.stdout: "ro_compat": {}, 2024-09-13T12:16:18.294 INFO:tasks.workunit.client.0.smithi032.stdout: "incompat": { 2024-09-13T12:16:18.294 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_1": "base v0.20", 2024-09-13T12:16:18.294 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_2": "client writeable ranges", 2024-09-13T12:16:18.294 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_3": "default file layouts on dirs", 2024-09-13T12:16:18.294 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_4": "dir inode in separate object", 2024-09-13T12:16:18.294 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_5": "mds uses versioned encoding", 2024-09-13T12:16:18.294 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-13T12:16:18.294 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_8": "no anchor table", 2024-09-13T12:16:18.294 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_9": "file layout v2", 2024-09-13T12:16:18.295 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_10": "snaprealm v2", 2024-09-13T12:16:18.295 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_11": "minor log segments", 2024-09-13T12:16:18.295 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_12": "quiesce subvolumes" 2024-09-13T12:16:18.295 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.295 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.295 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_flags": { 2024-09-13T12:16:18.295 INFO:tasks.workunit.client.0.smithi032.stdout: "enable_multiple": true, 2024-09-13T12:16:18.295 INFO:tasks.workunit.client.0.smithi032.stdout: "ever_enabled_multiple": true 2024-09-13T12:16:18.295 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.295 INFO:tasks.workunit.client.0.smithi032.stdout: "standbys": [], 2024-09-13T12:16:18.295 INFO:tasks.workunit.client.0.smithi032.stdout: "filesystems": [] 2024-09-13T12:16:18.295 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.295 INFO:tasks.workunit.client.0.smithi032.stdout: "mdsmap_first_committed": 1, 2024-09-13T12:16:18.295 INFO:tasks.workunit.client.0.smithi032.stdout: "mdsmap_last_committed": 1, 2024-09-13T12:16:18.295 INFO:tasks.workunit.client.0.smithi032.stdout: "auth": { 2024-09-13T12:16:18.296 INFO:tasks.workunit.client.0.smithi032.stdout: "first_committed": 1, 2024-09-13T12:16:18.296 INFO:tasks.workunit.client.0.smithi032.stdout: "last_committed": 5, 2024-09-13T12:16:18.296 INFO:tasks.workunit.client.0.smithi032.stdout: "num_secrets": 3 2024-09-13T12:16:18.296 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.296 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg": 1, 2024-09-13T12:16:18.296 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg_active": 1, 2024-09-13T12:16:18.296 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg_unknown": 0, 2024-09-13T12:16:18.296 INFO:tasks.workunit.client.0.smithi032.stdout: "num_osd": 3, 2024-09-13T12:16:18.296 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_sum": { 2024-09-13T12:16:18.296 INFO:tasks.workunit.client.0.smithi032.stdout: "stat_sum": { 2024-09-13T12:16:18.296 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes": 48454, 2024-09-13T12:16:18.296 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects": 101, 2024-09-13T12:16:18.296 INFO:tasks.workunit.client.0.smithi032.stdout: "num_object_clones": 1, 2024-09-13T12:16:18.296 INFO:tasks.workunit.client.0.smithi032.stdout: "num_object_copies": 303, 2024-09-13T12:16:18.297 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_missing_on_primary": 0, 2024-09-13T12:16:18.297 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_missing": 0, 2024-09-13T12:16:18.297 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_degraded": 101, 2024-09-13T12:16:18.297 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_misplaced": 0, 2024-09-13T12:16:18.297 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_unfound": 0, 2024-09-13T12:16:18.297 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_dirty": 101, 2024-09-13T12:16:18.297 INFO:tasks.workunit.client.0.smithi032.stdout: "num_whiteouts": 0, 2024-09-13T12:16:18.297 INFO:tasks.workunit.client.0.smithi032.stdout: "num_read": 0, 2024-09-13T12:16:18.297 INFO:tasks.workunit.client.0.smithi032.stdout: "num_read_kb": 0, 2024-09-13T12:16:18.297 INFO:tasks.workunit.client.0.smithi032.stdout: "num_write": 101, 2024-09-13T12:16:18.297 INFO:tasks.workunit.client.0.smithi032.stdout: "num_write_kb": 101, 2024-09-13T12:16:18.297 INFO:tasks.workunit.client.0.smithi032.stdout: "num_scrub_errors": 0, 2024-09-13T12:16:18.297 INFO:tasks.workunit.client.0.smithi032.stdout: "num_shallow_scrub_errors": 0, 2024-09-13T12:16:18.297 INFO:tasks.workunit.client.0.smithi032.stdout: "num_deep_scrub_errors": 0, 2024-09-13T12:16:18.297 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_recovered": 0, 2024-09-13T12:16:18.298 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes_recovered": 0, 2024-09-13T12:16:18.298 INFO:tasks.workunit.client.0.smithi032.stdout: "num_keys_recovered": 0, 2024-09-13T12:16:18.298 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_omap": 0, 2024-09-13T12:16:18.298 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_hit_set_archive": 0, 2024-09-13T12:16:18.298 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes_hit_set_archive": 0, 2024-09-13T12:16:18.298 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush": 0, 2024-09-13T12:16:18.298 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_kb": 0, 2024-09-13T12:16:18.298 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict": 0, 2024-09-13T12:16:18.298 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_kb": 0, 2024-09-13T12:16:18.298 INFO:tasks.workunit.client.0.smithi032.stdout: "num_promote": 0, 2024-09-13T12:16:18.298 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_mode_high": 0, 2024-09-13T12:16:18.298 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_mode_low": 0, 2024-09-13T12:16:18.298 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_mode_some": 0, 2024-09-13T12:16:18.298 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_mode_full": 0, 2024-09-13T12:16:18.299 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_pinned": 0, 2024-09-13T12:16:18.299 INFO:tasks.workunit.client.0.smithi032.stdout: "num_legacy_snapsets": 0, 2024-09-13T12:16:18.299 INFO:tasks.workunit.client.0.smithi032.stdout: "num_large_omap_objects": 0, 2024-09-13T12:16:18.299 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_manifest": 0, 2024-09-13T12:16:18.299 INFO:tasks.workunit.client.0.smithi032.stdout: "num_omap_bytes": 0, 2024-09-13T12:16:18.299 INFO:tasks.workunit.client.0.smithi032.stdout: "num_omap_keys": 0, 2024-09-13T12:16:18.299 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_repaired": 0 2024-09-13T12:16:18.299 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.299 INFO:tasks.workunit.client.0.smithi032.stdout: "store_stats": { 2024-09-13T12:16:18.299 INFO:tasks.workunit.client.0.smithi032.stdout: "total": 0, 2024-09-13T12:16:18.299 INFO:tasks.workunit.client.0.smithi032.stdout: "available": 0, 2024-09-13T12:16:18.299 INFO:tasks.workunit.client.0.smithi032.stdout: "internally_reserved": 0, 2024-09-13T12:16:18.299 INFO:tasks.workunit.client.0.smithi032.stdout: "allocated": 0, 2024-09-13T12:16:18.299 INFO:tasks.workunit.client.0.smithi032.stdout: "data_stored": 0, 2024-09-13T12:16:18.300 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed": 0, 2024-09-13T12:16:18.300 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_allocated": 0, 2024-09-13T12:16:18.300 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_original": 0, 2024-09-13T12:16:18.300 INFO:tasks.workunit.client.0.smithi032.stdout: "omap_allocated": 0, 2024-09-13T12:16:18.300 INFO:tasks.workunit.client.0.smithi032.stdout: "internal_metadata": 0 2024-09-13T12:16:18.300 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.300 INFO:tasks.workunit.client.0.smithi032.stdout: "log_size": 18, 2024-09-13T12:16:18.300 INFO:tasks.workunit.client.0.smithi032.stdout: "ondisk_log_size": 18, 2024-09-13T12:16:18.300 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 3, 2024-09-13T12:16:18.300 INFO:tasks.workunit.client.0.smithi032.stdout: "acting": 3, 2024-09-13T12:16:18.300 INFO:tasks.workunit.client.0.smithi032.stdout: "num_store_stats": 0 2024-09-13T12:16:18.300 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.300 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_sum": { 2024-09-13T12:16:18.300 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 0, 2024-09-13T12:16:18.301 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 0, 2024-09-13T12:16:18.301 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pgs": 3, 2024-09-13T12:16:18.301 INFO:tasks.workunit.client.0.smithi032.stdout: "num_osds": 3, 2024-09-13T12:16:18.301 INFO:tasks.workunit.client.0.smithi032.stdout: "num_per_pool_osds": 3, 2024-09-13T12:16:18.301 INFO:tasks.workunit.client.0.smithi032.stdout: "num_per_pool_omap_osds": 3, 2024-09-13T12:16:18.301 INFO:tasks.workunit.client.0.smithi032.stdout: "kb": 314572800, 2024-09-13T12:16:18.301 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used": 83676, 2024-09-13T12:16:18.301 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_data": 2148, 2024-09-13T12:16:18.301 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_omap": 10, 2024-09-13T12:16:18.301 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_meta": 81461, 2024-09-13T12:16:18.301 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_avail": 314489124, 2024-09-13T12:16:18.301 INFO:tasks.workunit.client.0.smithi032.stdout: "statfs": { 2024-09-13T12:16:18.301 INFO:tasks.workunit.client.0.smithi032.stdout: "total": 322122547200, 2024-09-13T12:16:18.301 INFO:tasks.workunit.client.0.smithi032.stdout: "available": 322036862976, 2024-09-13T12:16:18.302 INFO:tasks.workunit.client.0.smithi032.stdout: "internally_reserved": 0, 2024-09-13T12:16:18.302 INFO:tasks.workunit.client.0.smithi032.stdout: "allocated": 2199552, 2024-09-13T12:16:18.302 INFO:tasks.workunit.client.0.smithi032.stdout: "data_stored": 1544251, 2024-09-13T12:16:18.302 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed": 0, 2024-09-13T12:16:18.302 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_allocated": 0, 2024-09-13T12:16:18.302 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_original": 0, 2024-09-13T12:16:18.302 INFO:tasks.workunit.client.0.smithi032.stdout: "omap_allocated": 10396, 2024-09-13T12:16:18.302 INFO:tasks.workunit.client.0.smithi032.stdout: "internal_metadata": 83416932 2024-09-13T12:16:18.302 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.302 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_peers": [], 2024-09-13T12:16:18.302 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_trim_queue_len": 0, 2024-09-13T12:16:18.302 INFO:tasks.workunit.client.0.smithi032.stdout: "num_snap_trimming": 0, 2024-09-13T12:16:18.302 INFO:tasks.workunit.client.0.smithi032.stdout: "num_shards_repaired": 0, 2024-09-13T12:16:18.302 INFO:tasks.workunit.client.0.smithi032.stdout: "op_queue_age_hist": { 2024-09-13T12:16:18.303 INFO:tasks.workunit.client.0.smithi032.stdout: "histogram": [], 2024-09-13T12:16:18.303 INFO:tasks.workunit.client.0.smithi032.stdout: "upper_bound": 1 2024-09-13T12:16:18.303 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.303 INFO:tasks.workunit.client.0.smithi032.stdout: "perf_stat": { 2024-09-13T12:16:18.303 INFO:tasks.workunit.client.0.smithi032.stdout: "commit_latency_ms": 0, 2024-09-13T12:16:18.303 INFO:tasks.workunit.client.0.smithi032.stdout: "apply_latency_ms": 0, 2024-09-13T12:16:18.303 INFO:tasks.workunit.client.0.smithi032.stdout: "commit_latency_ns": 0, 2024-09-13T12:16:18.303 INFO:tasks.workunit.client.0.smithi032.stdout: "apply_latency_ns": 0 2024-09-13T12:16:18.303 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.303 INFO:tasks.workunit.client.0.smithi032.stdout: "alerts": [], 2024-09-13T12:16:18.303 INFO:tasks.workunit.client.0.smithi032.stdout: "network_ping_times": [] 2024-09-13T12:16:18.303 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.303 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_sum_by_class": { 2024-09-13T12:16:18.303 INFO:tasks.workunit.client.0.smithi032.stdout: "hdd": { 2024-09-13T12:16:18.304 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 0, 2024-09-13T12:16:18.304 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 0, 2024-09-13T12:16:18.304 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pgs": 3, 2024-09-13T12:16:18.304 INFO:tasks.workunit.client.0.smithi032.stdout: "num_osds": 3, 2024-09-13T12:16:18.304 INFO:tasks.workunit.client.0.smithi032.stdout: "num_per_pool_osds": 3, 2024-09-13T12:16:18.304 INFO:tasks.workunit.client.0.smithi032.stdout: "num_per_pool_omap_osds": 3, 2024-09-13T12:16:18.304 INFO:tasks.workunit.client.0.smithi032.stdout: "kb": 314572800, 2024-09-13T12:16:18.304 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used": 83676, 2024-09-13T12:16:18.304 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_data": 2148, 2024-09-13T12:16:18.304 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_omap": 10, 2024-09-13T12:16:18.304 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_meta": 81461, 2024-09-13T12:16:18.304 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_avail": 314489124, 2024-09-13T12:16:18.304 INFO:tasks.workunit.client.0.smithi032.stdout: "statfs": { 2024-09-13T12:16:18.304 INFO:tasks.workunit.client.0.smithi032.stdout: "total": 322122547200, 2024-09-13T12:16:18.305 INFO:tasks.workunit.client.0.smithi032.stdout: "available": 322036862976, 2024-09-13T12:16:18.305 INFO:tasks.workunit.client.0.smithi032.stdout: "internally_reserved": 0, 2024-09-13T12:16:18.305 INFO:tasks.workunit.client.0.smithi032.stdout: "allocated": 2199552, 2024-09-13T12:16:18.305 INFO:tasks.workunit.client.0.smithi032.stdout: "data_stored": 1544251, 2024-09-13T12:16:18.305 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed": 0, 2024-09-13T12:16:18.305 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_allocated": 0, 2024-09-13T12:16:18.305 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_original": 0, 2024-09-13T12:16:18.305 INFO:tasks.workunit.client.0.smithi032.stdout: "omap_allocated": 10396, 2024-09-13T12:16:18.305 INFO:tasks.workunit.client.0.smithi032.stdout: "internal_metadata": 83416932 2024-09-13T12:16:18.305 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.305 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_peers": [], 2024-09-13T12:16:18.305 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_trim_queue_len": 0, 2024-09-13T12:16:18.305 INFO:tasks.workunit.client.0.smithi032.stdout: "num_snap_trimming": 0, 2024-09-13T12:16:18.305 INFO:tasks.workunit.client.0.smithi032.stdout: "num_shards_repaired": 0, 2024-09-13T12:16:18.306 INFO:tasks.workunit.client.0.smithi032.stdout: "op_queue_age_hist": { 2024-09-13T12:16:18.306 INFO:tasks.workunit.client.0.smithi032.stdout: "histogram": [], 2024-09-13T12:16:18.306 INFO:tasks.workunit.client.0.smithi032.stdout: "upper_bound": 1 2024-09-13T12:16:18.306 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.306 INFO:tasks.workunit.client.0.smithi032.stdout: "perf_stat": { 2024-09-13T12:16:18.306 INFO:tasks.workunit.client.0.smithi032.stdout: "commit_latency_ms": 0, 2024-09-13T12:16:18.306 INFO:tasks.workunit.client.0.smithi032.stdout: "apply_latency_ms": 0, 2024-09-13T12:16:18.306 INFO:tasks.workunit.client.0.smithi032.stdout: "commit_latency_ns": 0, 2024-09-13T12:16:18.306 INFO:tasks.workunit.client.0.smithi032.stdout: "apply_latency_ns": 0 2024-09-13T12:16:18.306 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.306 INFO:tasks.workunit.client.0.smithi032.stdout: "alerts": [], 2024-09-13T12:16:18.306 INFO:tasks.workunit.client.0.smithi032.stdout: "network_ping_times": [] 2024-09-13T12:16:18.306 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.306 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.306 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_stats": [ 2024-09-13T12:16:18.307 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.307 INFO:tasks.workunit.client.0.smithi032.stdout: "poolid": 1, 2024-09-13T12:16:18.307 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg": 1, 2024-09-13T12:16:18.307 INFO:tasks.workunit.client.0.smithi032.stdout: "stat_sum": { 2024-09-13T12:16:18.307 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes": 48454, 2024-09-13T12:16:18.307 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects": 101, 2024-09-13T12:16:18.307 INFO:tasks.workunit.client.0.smithi032.stdout: "num_object_clones": 1, 2024-09-13T12:16:18.307 INFO:tasks.workunit.client.0.smithi032.stdout: "num_object_copies": 303, 2024-09-13T12:16:18.307 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_missing_on_primary": 0, 2024-09-13T12:16:18.307 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_missing": 0, 2024-09-13T12:16:18.307 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_degraded": 101, 2024-09-13T12:16:18.307 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_misplaced": 0, 2024-09-13T12:16:18.307 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_unfound": 0, 2024-09-13T12:16:18.307 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_dirty": 101, 2024-09-13T12:16:18.308 INFO:tasks.workunit.client.0.smithi032.stdout: "num_whiteouts": 0, 2024-09-13T12:16:18.308 INFO:tasks.workunit.client.0.smithi032.stdout: "num_read": 0, 2024-09-13T12:16:18.308 INFO:tasks.workunit.client.0.smithi032.stdout: "num_read_kb": 0, 2024-09-13T12:16:18.308 INFO:tasks.workunit.client.0.smithi032.stdout: "num_write": 101, 2024-09-13T12:16:18.308 INFO:tasks.workunit.client.0.smithi032.stdout: "num_write_kb": 101, 2024-09-13T12:16:18.308 INFO:tasks.workunit.client.0.smithi032.stdout: "num_scrub_errors": 0, 2024-09-13T12:16:18.308 INFO:tasks.workunit.client.0.smithi032.stdout: "num_shallow_scrub_errors": 0, 2024-09-13T12:16:18.308 INFO:tasks.workunit.client.0.smithi032.stdout: "num_deep_scrub_errors": 0, 2024-09-13T12:16:18.308 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_recovered": 0, 2024-09-13T12:16:18.308 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes_recovered": 0, 2024-09-13T12:16:18.308 INFO:tasks.workunit.client.0.smithi032.stdout: "num_keys_recovered": 0, 2024-09-13T12:16:18.308 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_omap": 0, 2024-09-13T12:16:18.308 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_hit_set_archive": 0, 2024-09-13T12:16:18.308 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes_hit_set_archive": 0, 2024-09-13T12:16:18.309 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush": 0, 2024-09-13T12:16:18.309 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_kb": 0, 2024-09-13T12:16:18.309 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict": 0, 2024-09-13T12:16:18.309 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_kb": 0, 2024-09-13T12:16:18.309 INFO:tasks.workunit.client.0.smithi032.stdout: "num_promote": 0, 2024-09-13T12:16:18.309 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_mode_high": 0, 2024-09-13T12:16:18.309 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_mode_low": 0, 2024-09-13T12:16:18.309 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_mode_some": 0, 2024-09-13T12:16:18.309 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_mode_full": 0, 2024-09-13T12:16:18.309 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_pinned": 0, 2024-09-13T12:16:18.309 INFO:tasks.workunit.client.0.smithi032.stdout: "num_legacy_snapsets": 0, 2024-09-13T12:16:18.309 INFO:tasks.workunit.client.0.smithi032.stdout: "num_large_omap_objects": 0, 2024-09-13T12:16:18.309 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_manifest": 0, 2024-09-13T12:16:18.309 INFO:tasks.workunit.client.0.smithi032.stdout: "num_omap_bytes": 0, 2024-09-13T12:16:18.310 INFO:tasks.workunit.client.0.smithi032.stdout: "num_omap_keys": 0, 2024-09-13T12:16:18.310 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_repaired": 0 2024-09-13T12:16:18.310 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.310 INFO:tasks.workunit.client.0.smithi032.stdout: "store_stats": { 2024-09-13T12:16:18.310 INFO:tasks.workunit.client.0.smithi032.stdout: "total": 0, 2024-09-13T12:16:18.310 INFO:tasks.workunit.client.0.smithi032.stdout: "available": 0, 2024-09-13T12:16:18.310 INFO:tasks.workunit.client.0.smithi032.stdout: "internally_reserved": 0, 2024-09-13T12:16:18.310 INFO:tasks.workunit.client.0.smithi032.stdout: "allocated": 1241088, 2024-09-13T12:16:18.310 INFO:tasks.workunit.client.0.smithi032.stdout: "data_stored": 1241088, 2024-09-13T12:16:18.310 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed": 0, 2024-09-13T12:16:18.310 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_allocated": 0, 2024-09-13T12:16:18.310 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_original": 0, 2024-09-13T12:16:18.310 INFO:tasks.workunit.client.0.smithi032.stdout: "omap_allocated": 0, 2024-09-13T12:16:18.310 INFO:tasks.workunit.client.0.smithi032.stdout: "internal_metadata": 0 2024-09-13T12:16:18.310 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.311 INFO:tasks.workunit.client.0.smithi032.stdout: "log_size": 18, 2024-09-13T12:16:18.311 INFO:tasks.workunit.client.0.smithi032.stdout: "ondisk_log_size": 18, 2024-09-13T12:16:18.311 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 3, 2024-09-13T12:16:18.311 INFO:tasks.workunit.client.0.smithi032.stdout: "acting": 3, 2024-09-13T12:16:18.311 INFO:tasks.workunit.client.0.smithi032.stdout: "num_store_stats": 3 2024-09-13T12:16:18.311 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.311 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:16:18.311 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_stats": [ 2024-09-13T12:16:18.311 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.311 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 0, 2024-09-13T12:16:18.311 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 158913789964 2024-09-13T12:16:18.311 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.311 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.311 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 1, 2024-09-13T12:16:18.312 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 42949672990 2024-09-13T12:16:18.312 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.312 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.312 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2, 2024-09-13T12:16:18.312 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 167503724555 2024-09-13T12:16:18.312 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.312 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:16:18.312 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg_by_state": [ 2024-09-13T12:16:18.312 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.312 INFO:tasks.workunit.client.0.smithi032.stdout: "state": "active+undersized+degraded", 2024-09-13T12:16:18.312 INFO:tasks.workunit.client.0.smithi032.stdout: "num": 1 2024-09-13T12:16:18.312 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.312 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:16:18.312 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg_by_osd": [ 2024-09-13T12:16:18.313 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.313 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2147483647, 2024-09-13T12:16:18.313 INFO:tasks.workunit.client.0.smithi032.stdout: "num_primary_pg": 0, 2024-09-13T12:16:18.313 INFO:tasks.workunit.client.0.smithi032.stdout: "num_acting_pg": 1, 2024-09-13T12:16:18.313 INFO:tasks.workunit.client.0.smithi032.stdout: "num_up_not_acting_pg": 0 2024-09-13T12:16:18.313 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.313 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.313 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2, 2024-09-13T12:16:18.313 INFO:tasks.workunit.client.0.smithi032.stdout: "num_primary_pg": 0, 2024-09-13T12:16:18.313 INFO:tasks.workunit.client.0.smithi032.stdout: "num_acting_pg": 1, 2024-09-13T12:16:18.313 INFO:tasks.workunit.client.0.smithi032.stdout: "num_up_not_acting_pg": 0 2024-09-13T12:16:18.313 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.313 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.313 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 0, 2024-09-13T12:16:18.313 INFO:tasks.workunit.client.0.smithi032.stdout: "num_primary_pg": 1, 2024-09-13T12:16:18.314 INFO:tasks.workunit.client.0.smithi032.stdout: "num_acting_pg": 1, 2024-09-13T12:16:18.314 INFO:tasks.workunit.client.0.smithi032.stdout: "num_up_not_acting_pg": 0 2024-09-13T12:16:18.314 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.314 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.314 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 1, 2024-09-13T12:16:18.314 INFO:tasks.workunit.client.0.smithi032.stdout: "num_primary_pg": 0, 2024-09-13T12:16:18.314 INFO:tasks.workunit.client.0.smithi032.stdout: "num_acting_pg": 0, 2024-09-13T12:16:18.314 INFO:tasks.workunit.client.0.smithi032.stdout: "num_up_not_acting_pg": 0 2024-09-13T12:16:18.314 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.314 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:16:18.314 INFO:tasks.workunit.client.0.smithi032.stdout: "purged_snaps": [ 2024-09-13T12:16:18.314 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:16:18.314 INFO:tasks.workunit.client.0.smithi032.stdout: "pool": 1, 2024-09-13T12:16:18.314 INFO:tasks.workunit.client.0.smithi032.stdout: "purged_snaps": {} 2024-09-13T12:16:18.314 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.315 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:16:18.315 INFO:tasks.workunit.client.0.smithi032.stdout: "servicemap": { 2024-09-13T12:16:18.315 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 5, 2024-09-13T12:16:18.315 INFO:tasks.workunit.client.0.smithi032.stdout: "modified": "2024-09-13T12:15:20.230759+0000", 2024-09-13T12:16:18.315 INFO:tasks.workunit.client.0.smithi032.stdout: "services": {} 2024-09-13T12:16:18.315 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:16:18.315 INFO:tasks.workunit.client.0.smithi032.stdout: "mgrstat_first_committed": 1, 2024-09-13T12:16:18.315 INFO:tasks.workunit.client.0.smithi032.stdout: "mgrstat_last_committed": 102, 2024-09-13T12:16:18.315 INFO:tasks.workunit.client.0.smithi032.stdout: "logm_first_committed": 1, 2024-09-13T12:16:18.315 INFO:tasks.workunit.client.0.smithi032.stdout: "logm_last_committed": 401, 2024-09-13T12:16:18.315 INFO:tasks.workunit.client.0.smithi032.stdout: "paxos": { 2024-09-13T12:16:18.315 INFO:tasks.workunit.client.0.smithi032.stdout: "first_committed": 1, 2024-09-13T12:16:18.315 INFO:tasks.workunit.client.0.smithi032.stdout: "last_committed": 554, 2024-09-13T12:16:18.315 INFO:tasks.workunit.client.0.smithi032.stdout: "last_pn": 0, 2024-09-13T12:16:18.315 INFO:tasks.workunit.client.0.smithi032.stdout: "accepted_pn": 0 2024-09-13T12:16:18.316 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:16:18.316 INFO:tasks.workunit.client.0.smithi032.stdout:} 2024-09-13T12:16:18.317 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-13T12:16:18.317 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-09-13T12:16:18.317 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2024-09-13T12:16:18.317 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:16:18.317 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:16:18.317 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:16:18.317 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:16:18.317 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:16:18.341 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:16:18.341 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2024-09-13T12:16:18.341 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-13T12:16:18.341 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T12:16:18.341 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T12:16:18.341 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T12:16:18.342 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2024-09-13T12:16:18.342 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2024-09-13T12:16:19.768 INFO:tasks.workunit.client.0.smithi032.stderr:PG '1.0' not found 2024-09-13T12:16:20.038 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1256: _objectstore_tool_nodown: return 1 2024-09-13T12:16:20.038 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-09-13T12:16:20.039 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-13T12:16:20.039 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:16:20.039 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T12:16:20.039 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:16:20.039 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/0 2024-09-13T12:16:20.039 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T12:16:20.039 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:16:20.039 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:16:20.039 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:16:20.039 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-13T12:16:20.039 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-13T12:16:20.039 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:16:20.040 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:16:20.040 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T12:16:20.040 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:16:20.040 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:16:20.040 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:16:20.040 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:16:20.040 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:16:20.040 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:16:20.040 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:16:20.040 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:16:20.040 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:16:20.040 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T12:16:20.040 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T12:16:20.041 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:16:20.041 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:16:20.041 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:16:20.041 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:16:20.041 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:16:20.041 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:16:20.041 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/0 2024-09-13T12:16:20.041 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.0 2024-09-13T12:16:20.042 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T12:16:20.042 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:16:20.042 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/0/whoami 2024-09-13T12:16:20.042 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T12:16:20.043 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:16:20.043 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:16:20.043 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:16:20.079 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:16:20.078+0000 7fa836631640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:16:20.081 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:16:20.081+0000 7fa836631640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:16:20.083 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:16:20.082+0000 7fa836631640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:16:20.493 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T12:16:20.494 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:16:20.494 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T12:16:20.494 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:16:20.494 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:16:20.494 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:16:20.494 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:16:20.494 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:16:20.495 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:16:20.495 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:16:20.904 INFO:tasks.workunit.client.0.smithi032.stdout:osd.0 up in weight 1 up_from 37 up_thru 41 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/4196072543,v1:127.0.0.1:6803/4196072543] [v2:127.0.0.1:6804/4196072543,v1:127.0.0.1:6805/4196072543] exists,up 92bce19c-5a5e-4360-a088-1bf366fc2ecf 2024-09-13T12:16:20.906 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:16:20.906 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:16:20.906 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:16:20.906 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-09-13T12:16:20.906 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2024-09-13T12:16:20.906 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:16:20.907 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:16:20.907 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:16:20.907 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:16:20.907 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:16:21.011 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:16:21.011 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2024-09-13T12:16:21.011 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-13T12:16:21.012 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T12:16:21.012 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-13T12:16:21.012 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T12:16:21.012 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2024-09-13T12:16:21.012 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2024-09-13T12:16:21.444 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:16:21.443+0000 7fa836631640 -1 Falling back to public interface 2024-09-13T12:16:22.429 INFO:tasks.workunit.client.0.smithi032.stderr:PG '1.0' not found 2024-09-13T12:16:22.700 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1256: _objectstore_tool_nodown: return 1 2024-09-13T12:16:22.700 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-09-13T12:16:22.700 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-13T12:16:22.700 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:16:22.700 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-13T12:16:22.700 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:16:22.700 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/2 2024-09-13T12:16:22.701 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T12:16:22.701 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:16:22.701 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:16:22.701 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:16:22.701 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-13T12:16:22.701 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-13T12:16:22.701 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:16:22.701 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:16:22.701 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T12:16:22.701 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:16:22.701 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:16:22.701 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:16:22.701 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:16:22.702 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:16:22.702 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:16:22.702 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:16:22.702 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:16:22.702 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:16:22.702 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T12:16:22.702 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T12:16:22.702 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:16:22.702 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:16:22.702 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:16:22.702 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:16:22.702 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:16:22.702 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:16:22.702 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/2 2024-09-13T12:16:22.703 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.2 2024-09-13T12:16:22.704 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-13T12:16:22.704 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:16:22.704 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/2/whoami 2024-09-13T12:16:22.704 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-13T12:16:22.704 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:16:22.704 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:16:22.704 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:16:22.741 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:16:22.740+0000 7f14cf422640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:16:22.743 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:16:22.742+0000 7f14cf422640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:16:22.744 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:16:22.744+0000 7f14cf422640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:16:23.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-13T12:16:23.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:16:23.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T12:16:23.159 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:16:23.159 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:16:23.159 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:16:23.159 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:16:23.159 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:16:23.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:16:23.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:16:23.601 INFO:tasks.workunit.client.0.smithi032.stdout:osd.2 up in weight 1 up_from 39 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/3815047400,v1:127.0.0.1:6811/3815047400] [v2:127.0.0.1:6812/3815047400,v1:127.0.0.1:6813/3815047400] exists,up 575c91a2-dd41-4141-bbec-f3711b8a1a2e 2024-09-13T12:16:23.602 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:16:23.602 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:16:23.602 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:16:23.602 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2024-09-13T12:16:23.602 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-13T12:16:23.602 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T12:16:23.602 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T12:16:23.602 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T12:16:23.602 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-13T12:16:23.602 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2024-09-13T12:16:24.106 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:16:24.105+0000 7fa836631640 -1 osd.0 42 log_to_monitors true 2024-09-13T12:16:24.344 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:16:24.343+0000 7f14cf422640 -1 Falling back to public interface 2024-09-13T12:16:25.956 INFO:tasks.workunit.client.0.smithi032.stderr:PG '1.0' not found 2024-09-13T12:16:26.222 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1256: _objectstore_tool_nodown: return 1 2024-09-13T12:16:26.222 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-13T12:16:26.222 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2024-09-13T12:16:26.222 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T12:16:26.222 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T12:16:26.222 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T12:16:26.222 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-13T12:16:26.223 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T12:16:26.223 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T12:16:26.223 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T12:16:26.223 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T12:16:26.223 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T12:16:26.289 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-13T12:16:26.289 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T12:16:26.289 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T12:16:26.289 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T12:16:26.289 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T12:16:26.290 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T12:16:26.750 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T12:16:26.750 INFO:tasks.workunit.client.0.smithi032.stderr:1 2024-09-13T12:16:26.750 INFO:tasks.workunit.client.0.smithi032.stderr:2' 2024-09-13T12:16:26.750 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T12:16:26.750 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:16:26.751 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T12:16:26.889 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528323 2024-09-13T12:16:26.890 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528323 2024-09-13T12:16:26.890 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-193273528323' 2024-09-13T12:16:26.890 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:16:26.890 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T12:16:26.960 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:16:26.959+0000 7f14cf422640 -1 osd.2 42 log_to_monitors true 2024-09-13T12:16:27.039 INFO:tasks.workunit.client.0.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-13T12:16:27.041 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-13T12:16:27.041 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-13T12:16:27.041 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-13T12:16:27.042 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:16:27.042 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T12:16:32.171 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=210453397507 2024-09-13T12:16:32.171 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 210453397507 2024-09-13T12:16:32.171 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-193273528323 2-210453397507' 2024-09-13T12:16:32.172 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:16:32.172 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-193273528323 2024-09-13T12:16:32.172 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:16:32.173 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T12:16:32.174 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-193273528323 2024-09-13T12:16:32.174 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:16:32.175 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528323 2024-09-13T12:16:32.175 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.0 seq 193273528323 2024-09-13T12:16:32.175 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 193273528323' 2024-09-13T12:16:32.176 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:16:32.616 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528324 -lt 193273528323 2024-09-13T12:16:32.616 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:16:32.616 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-210453397507 2024-09-13T12:16:32.617 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:16:32.617 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T12:16:32.618 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-210453397507 2024-09-13T12:16:32.618 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:16:32.619 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=210453397507 2024-09-13T12:16:32.619 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 210453397507' 2024-09-13T12:16:32.619 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.2 seq 210453397507 2024-09-13T12:16:32.619 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T12:16:33.061 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 210453397507 -lt 210453397507 2024-09-13T12:16:33.061 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T12:16:33.062 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:16:33.062 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:16:33.622 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T12:16:33.622 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:16:33.622 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:16:33.622 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:16:33.623 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:16:33.623 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:16:33.623 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:16:33.623 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:16:34.018 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:16:34.018 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:16:34.019 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:16:34.019 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:16:34.582 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:16:34.582 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-13T12:16:34.583 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T12:16:34.583 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-13T12:16:34.583 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:16:34.583 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T12:16:34.683 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:16:34.683 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:16:34.684 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:16:34.684 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:16:34.684 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:16:34.684 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:16:34.685 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:16:34.685 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:16:35.085 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:16:35.086 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:16:35.086 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:16:35.086 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:16:35.644 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:16:35.645 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:16:35.645 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:16:35.645 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:16:35.645 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:16:35.645 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:16:35.645 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:16:35.645 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:16:35.645 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:16:36.235 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:16:36.235 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:16:36.235 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-13T12:16:36.235 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:16:36.235 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T12:16:36.436 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:16:36.436 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:16:36.437 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:16:36.437 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:16:36.437 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:16:36.437 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:16:36.437 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:16:36.437 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:16:36.840 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:16:36.841 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:16:36.841 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:16:36.841 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:16:37.401 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:16:37.401 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:16:37.402 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:16:37.402 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:16:37.402 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:16:37.402 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:16:37.402 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:16:37.402 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:16:37.402 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:16:37.984 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:16:37.984 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:16:37.985 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-13T12:16:37.985 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:16:37.985 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-13T12:16:38.384 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:16:38.384 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:16:38.384 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:16:38.384 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:16:38.385 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:16:38.385 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:16:38.385 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:16:38.385 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:16:38.789 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:16:38.789 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:16:38.790 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:16:38.790 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:16:39.343 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:16:39.343 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:16:39.343 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:16:39.343 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:16:39.343 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:16:39.343 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:16:39.344 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:16:39.344 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:16:39.344 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:16:39.913 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:16:39.914 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:16:39.914 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-13T12:16:39.914 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:16:39.914 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-13T12:16:40.714 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:16:40.714 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:16:40.715 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:16:40.715 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:16:40.715 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:16:40.715 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:16:40.715 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:16:40.715 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:16:41.118 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:16:41.119 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:16:41.119 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:16:41.119 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:16:41.675 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:16:41.676 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:16:41.676 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:16:41.676 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:16:41.676 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:16:41.676 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:16:41.676 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:16:41.676 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:16:41.676 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:16:42.251 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:16:42.251 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:16:42.251 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-13T12:16:42.251 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:16:42.251 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-13T12:16:43.851 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:16:43.851 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:16:43.852 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:16:43.852 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:16:43.852 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:16:43.852 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:16:43.853 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:16:43.853 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:16:44.265 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:16:44.265 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:16:44.266 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:16:44.266 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:16:44.824 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:16:44.824 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:16:44.824 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:16:44.824 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:16:44.824 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:16:44.824 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:16:44.824 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:16:44.824 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:16:44.824 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:16:45.439 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:16:45.439 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:16:45.439 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-13T12:16:45.439 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:16:45.439 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-13T12:16:48.640 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:16:48.640 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:16:48.641 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:16:48.641 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:16:48.641 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:16:48.641 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:16:48.641 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:16:48.641 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:16:49.058 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:16:49.059 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:16:49.059 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:16:49.059 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:16:49.619 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:16:49.619 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:16:49.620 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:16:49.620 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:16:49.620 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:16:49.620 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:16:49.620 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:16:49.620 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:16:49.620 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:16:50.174 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:16:50.174 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:16:50.175 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-13T12:16:50.175 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:16:50.175 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-13T12:16:56.576 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:16:56.576 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:16:56.577 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:16:56.577 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:16:56.577 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:16:56.577 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:16:56.577 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:16:56.577 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:16:56.991 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:16:56.992 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:16:56.992 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:16:56.992 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:16:57.550 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:16:57.550 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:16:57.550 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:16:57.550 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:16:57.550 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:16:57.550 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:16:57.551 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:16:57.551 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:16:57.551 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:16:58.106 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:16:58.106 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:16:58.106 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-13T12:16:58.106 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T12:16:58.106 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-13T12:17:05.406 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T12:17:05.407 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:17:05.408 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:17:05.408 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:17:05.408 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:17:05.408 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:17:05.408 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:17:05.408 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:17:05.822 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T12:17:05.823 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:17:05.823 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:17:05.823 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:17:06.426 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T12:17:06.426 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T12:17:06.426 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T12:17:06.426 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T12:17:06.426 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T12:17:06.426 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T12:17:06.426 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T12:17:06.426 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T12:17:06.427 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T12:17:06.978 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T12:17:06.979 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T12:17:06.979 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-13T12:17:06.979 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-13T12:17:07.535 INFO:tasks.workunit.client.0.smithi032.stdout:{ 2024-09-13T12:17:07.535 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_fingerprint": "6ab5cc67-92b9-437c-9d83-cc24da6da1c9", 2024-09-13T12:17:07.535 INFO:tasks.workunit.client.0.smithi032.stdout: "version": "19.3.0-4697-gd58f7306", 2024-09-13T12:17:07.535 INFO:tasks.workunit.client.0.smithi032.stdout: "commit": "d58f73069a8157a9ace12a0247ee3e8497707ea9", 2024-09-13T12:17:07.535 INFO:tasks.workunit.client.0.smithi032.stdout: "timestamp": "2024-09-13T12:17:07.533866+0000", 2024-09-13T12:17:07.535 INFO:tasks.workunit.client.0.smithi032.stdout: "tag": "", 2024-09-13T12:17:07.535 INFO:tasks.workunit.client.0.smithi032.stdout: "health": { 2024-09-13T12:17:07.535 INFO:tasks.workunit.client.0.smithi032.stdout: "status": "HEALTH_WARN", 2024-09-13T12:17:07.535 INFO:tasks.workunit.client.0.smithi032.stdout: "checks": { 2024-09-13T12:17:07.535 INFO:tasks.workunit.client.0.smithi032.stdout: "OSDMAP_FLAGS": { 2024-09-13T12:17:07.536 INFO:tasks.workunit.client.0.smithi032.stdout: "severity": "HEALTH_WARN", 2024-09-13T12:17:07.536 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": { 2024-09-13T12:17:07.536 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-13T12:17:07.536 INFO:tasks.workunit.client.0.smithi032.stdout: "count": 17 2024-09-13T12:17:07.536 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.536 INFO:tasks.workunit.client.0.smithi032.stdout: "detail": [], 2024-09-13T12:17:07.536 INFO:tasks.workunit.client.0.smithi032.stdout: "muted": false 2024-09-13T12:17:07.536 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.536 INFO:tasks.workunit.client.0.smithi032.stdout: "OSD_DOWN": { 2024-09-13T12:17:07.536 INFO:tasks.workunit.client.0.smithi032.stdout: "severity": "HEALTH_WARN", 2024-09-13T12:17:07.536 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": { 2024-09-13T12:17:07.537 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "1 osds down", 2024-09-13T12:17:07.537 INFO:tasks.workunit.client.0.smithi032.stdout: "count": 1 2024-09-13T12:17:07.537 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.537 INFO:tasks.workunit.client.0.smithi032.stdout: "detail": [ 2024-09-13T12:17:07.537 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.537 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "osd.1 (root=default,host=smithi032) is down" 2024-09-13T12:17:07.537 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.537 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:17:07.537 INFO:tasks.workunit.client.0.smithi032.stdout: "muted": false 2024-09-13T12:17:07.537 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.537 INFO:tasks.workunit.client.0.smithi032.stdout: "PG_DEGRADED": { 2024-09-13T12:17:07.537 INFO:tasks.workunit.client.0.smithi032.stdout: "severity": "HEALTH_WARN", 2024-09-13T12:17:07.537 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": { 2024-09-13T12:17:07.537 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-09-13T12:17:07.538 INFO:tasks.workunit.client.0.smithi032.stdout: "count": 1 2024-09-13T12:17:07.538 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.538 INFO:tasks.workunit.client.0.smithi032.stdout: "detail": [ 2024-09-13T12:17:07.538 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.538 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-09-13T12:17:07.538 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.538 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:17:07.538 INFO:tasks.workunit.client.0.smithi032.stdout: "muted": false 2024-09-13T12:17:07.538 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.538 INFO:tasks.workunit.client.0.smithi032.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-13T12:17:07.538 INFO:tasks.workunit.client.0.smithi032.stdout: "severity": "HEALTH_WARN", 2024-09-13T12:17:07.539 INFO:tasks.workunit.client.0.smithi032.stdout: "summary": { 2024-09-13T12:17:07.539 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-13T12:17:07.539 INFO:tasks.workunit.client.0.smithi032.stdout: "count": 1 2024-09-13T12:17:07.540 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.540 INFO:tasks.workunit.client.0.smithi032.stdout: "detail": [ 2024-09-13T12:17:07.540 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.540 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "application not enabled on pool 'test'" 2024-09-13T12:17:07.540 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.540 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.540 INFO:tasks.workunit.client.0.smithi032.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-13T12:17:07.540 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.540 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:17:07.540 INFO:tasks.workunit.client.0.smithi032.stdout: "muted": false 2024-09-13T12:17:07.540 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.541 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.541 INFO:tasks.workunit.client.0.smithi032.stdout: "mutes": [] 2024-09-13T12:17:07.541 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.541 INFO:tasks.workunit.client.0.smithi032.stdout: "monmap_first_committed": 1, 2024-09-13T12:17:07.541 INFO:tasks.workunit.client.0.smithi032.stdout: "monmap_last_committed": 1, 2024-09-13T12:17:07.541 INFO:tasks.workunit.client.0.smithi032.stdout: "monmap": { 2024-09-13T12:17:07.541 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 1, 2024-09-13T12:17:07.541 INFO:tasks.workunit.client.0.smithi032.stdout: "fsid": "e12dfbc3-9bf5-45db-a6f0-f25db93ee351", 2024-09-13T12:17:07.541 INFO:tasks.workunit.client.0.smithi032.stdout: "modified": "2024-09-13T12:12:30.364120Z", 2024-09-13T12:17:07.541 INFO:tasks.workunit.client.0.smithi032.stdout: "created": "2024-09-13T12:12:30.364120Z", 2024-09-13T12:17:07.541 INFO:tasks.workunit.client.0.smithi032.stdout: "min_mon_release": 19, 2024-09-13T12:17:07.541 INFO:tasks.workunit.client.0.smithi032.stdout: "min_mon_release_name": "squid", 2024-09-13T12:17:07.541 INFO:tasks.workunit.client.0.smithi032.stdout: "election_strategy": 1, 2024-09-13T12:17:07.541 INFO:tasks.workunit.client.0.smithi032.stdout: "disallowed_leaders: ": "", 2024-09-13T12:17:07.542 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_mode": false, 2024-09-13T12:17:07.542 INFO:tasks.workunit.client.0.smithi032.stdout: "tiebreaker_mon": "", 2024-09-13T12:17:07.542 INFO:tasks.workunit.client.0.smithi032.stdout: "removed_ranks: ": "", 2024-09-13T12:17:07.542 INFO:tasks.workunit.client.0.smithi032.stdout: "features": { 2024-09-13T12:17:07.542 INFO:tasks.workunit.client.0.smithi032.stdout: "persistent": [ 2024-09-13T12:17:07.542 INFO:tasks.workunit.client.0.smithi032.stdout: "kraken", 2024-09-13T12:17:07.542 INFO:tasks.workunit.client.0.smithi032.stdout: "luminous", 2024-09-13T12:17:07.542 INFO:tasks.workunit.client.0.smithi032.stdout: "mimic", 2024-09-13T12:17:07.542 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap-prune", 2024-09-13T12:17:07.542 INFO:tasks.workunit.client.0.smithi032.stdout: "nautilus", 2024-09-13T12:17:07.542 INFO:tasks.workunit.client.0.smithi032.stdout: "octopus", 2024-09-13T12:17:07.542 INFO:tasks.workunit.client.0.smithi032.stdout: "pacific", 2024-09-13T12:17:07.542 INFO:tasks.workunit.client.0.smithi032.stdout: "elector-pinging", 2024-09-13T12:17:07.543 INFO:tasks.workunit.client.0.smithi032.stdout: "quincy", 2024-09-13T12:17:07.543 INFO:tasks.workunit.client.0.smithi032.stdout: "reef", 2024-09-13T12:17:07.543 INFO:tasks.workunit.client.0.smithi032.stdout: "squid" 2024-09-13T12:17:07.543 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:17:07.543 INFO:tasks.workunit.client.0.smithi032.stdout: "optional": [] 2024-09-13T12:17:07.543 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.543 INFO:tasks.workunit.client.0.smithi032.stdout: "mons": [ 2024-09-13T12:17:07.543 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.543 INFO:tasks.workunit.client.0.smithi032.stdout: "rank": 0, 2024-09-13T12:17:07.543 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "a", 2024-09-13T12:17:07.543 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addrs": { 2024-09-13T12:17:07.543 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:17:07.543 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.543 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:17:07.544 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:7115", 2024-09-13T12:17:07.544 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 0 2024-09-13T12:17:07.544 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.544 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:17:07.544 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.544 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:7115/0", 2024-09-13T12:17:07.544 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-13T12:17:07.544 INFO:tasks.workunit.client.0.smithi032.stdout: "priority": 0, 2024-09-13T12:17:07.544 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 0, 2024-09-13T12:17:07.544 INFO:tasks.workunit.client.0.smithi032.stdout: "crush_location": "{}" 2024-09-13T12:17:07.544 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.544 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:17:07.544 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.545 INFO:tasks.workunit.client.0.smithi032.stdout: "quorum": [ 2024-09-13T12:17:07.545 INFO:tasks.workunit.client.0.smithi032.stdout: 0 2024-09-13T12:17:07.545 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:17:07.545 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap": { 2024-09-13T12:17:07.545 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 50, 2024-09-13T12:17:07.545 INFO:tasks.workunit.client.0.smithi032.stdout: "fsid": "e12dfbc3-9bf5-45db-a6f0-f25db93ee351", 2024-09-13T12:17:07.545 INFO:tasks.workunit.client.0.smithi032.stdout: "created": "2024-09-13T12:12:31.284734+0000", 2024-09-13T12:17:07.545 INFO:tasks.workunit.client.0.smithi032.stdout: "modified": "2024-09-13T12:16:27.999451+0000", 2024-09-13T12:17:07.545 INFO:tasks.workunit.client.0.smithi032.stdout: "last_up_change": "2024-09-13T12:16:27.511337+0000", 2024-09-13T12:17:07.545 INFO:tasks.workunit.client.0.smithi032.stdout: "last_in_change": "2024-09-13T12:15:34.847471+0000", 2024-09-13T12:17:07.545 INFO:tasks.workunit.client.0.smithi032.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T12:17:07.545 INFO:tasks.workunit.client.0.smithi032.stdout: "flags_num": 5800384, 2024-09-13T12:17:07.545 INFO:tasks.workunit.client.0.smithi032.stdout: "flags_set": [ 2024-09-13T12:17:07.546 INFO:tasks.workunit.client.0.smithi032.stdout: "nodown", 2024-09-13T12:17:07.546 INFO:tasks.workunit.client.0.smithi032.stdout: "noin", 2024-09-13T12:17:07.546 INFO:tasks.workunit.client.0.smithi032.stdout: "noout", 2024-09-13T12:17:07.546 INFO:tasks.workunit.client.0.smithi032.stdout: "pglog_hardlimit", 2024-09-13T12:17:07.546 INFO:tasks.workunit.client.0.smithi032.stdout: "purged_snapdirs", 2024-09-13T12:17:07.546 INFO:tasks.workunit.client.0.smithi032.stdout: "recovery_deletes", 2024-09-13T12:17:07.546 INFO:tasks.workunit.client.0.smithi032.stdout: "sortbitwise" 2024-09-13T12:17:07.546 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:17:07.546 INFO:tasks.workunit.client.0.smithi032.stdout: "crush_version": 12, 2024-09-13T12:17:07.546 INFO:tasks.workunit.client.0.smithi032.stdout: "full_ratio": 0.99000000953674316, 2024-09-13T12:17:07.546 INFO:tasks.workunit.client.0.smithi032.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-13T12:17:07.546 INFO:tasks.workunit.client.0.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T12:17:07.546 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_snapshot": "", 2024-09-13T12:17:07.547 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_max": 1, 2024-09-13T12:17:07.547 INFO:tasks.workunit.client.0.smithi032.stdout: "max_osd": 3, 2024-09-13T12:17:07.547 INFO:tasks.workunit.client.0.smithi032.stdout: "require_min_compat_client": "luminous", 2024-09-13T12:17:07.547 INFO:tasks.workunit.client.0.smithi032.stdout: "min_compat_client": "jewel", 2024-09-13T12:17:07.547 INFO:tasks.workunit.client.0.smithi032.stdout: "require_osd_release": "squid", 2024-09-13T12:17:07.547 INFO:tasks.workunit.client.0.smithi032.stdout: "allow_crimson": false, 2024-09-13T12:17:07.547 INFO:tasks.workunit.client.0.smithi032.stdout: "pools": [ 2024-09-13T12:17:07.547 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.547 INFO:tasks.workunit.client.0.smithi032.stdout: "pool": 1, 2024-09-13T12:17:07.547 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_name": "test", 2024-09-13T12:17:07.547 INFO:tasks.workunit.client.0.smithi032.stdout: "create_time": "2024-09-13T12:14:22.654401+0000", 2024-09-13T12:17:07.547 INFO:tasks.workunit.client.0.smithi032.stdout: "flags": 16389, 2024-09-13T12:17:07.547 INFO:tasks.workunit.client.0.smithi032.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-09-13T12:17:07.547 INFO:tasks.workunit.client.0.smithi032.stdout: "type": 3, 2024-09-13T12:17:07.548 INFO:tasks.workunit.client.0.smithi032.stdout: "size": 3, 2024-09-13T12:17:07.548 INFO:tasks.workunit.client.0.smithi032.stdout: "min_size": 2, 2024-09-13T12:17:07.548 INFO:tasks.workunit.client.0.smithi032.stdout: "crush_rule": 1, 2024-09-13T12:17:07.548 INFO:tasks.workunit.client.0.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-13T12:17:07.548 INFO:tasks.workunit.client.0.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-13T12:17:07.548 INFO:tasks.workunit.client.0.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T12:17:07.548 INFO:tasks.workunit.client.0.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T12:17:07.548 INFO:tasks.workunit.client.0.smithi032.stdout: "is_stretch_pool": false, 2024-09-13T12:17:07.548 INFO:tasks.workunit.client.0.smithi032.stdout: "object_hash": 2, 2024-09-13T12:17:07.548 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_autoscale_mode": "off", 2024-09-13T12:17:07.548 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_num": 1, 2024-09-13T12:17:07.548 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_placement_num": 1, 2024-09-13T12:17:07.548 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_placement_num_target": 1, 2024-09-13T12:17:07.548 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_num_target": 1, 2024-09-13T12:17:07.549 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_num_pending": 1, 2024-09-13T12:17:07.549 INFO:tasks.workunit.client.0.smithi032.stdout: "last_pg_merge_meta": { 2024-09-13T12:17:07.549 INFO:tasks.workunit.client.0.smithi032.stdout: "source_pgid": "0.0", 2024-09-13T12:17:07.549 INFO:tasks.workunit.client.0.smithi032.stdout: "ready_epoch": 0, 2024-09-13T12:17:07.549 INFO:tasks.workunit.client.0.smithi032.stdout: "last_epoch_started": 0, 2024-09-13T12:17:07.549 INFO:tasks.workunit.client.0.smithi032.stdout: "last_epoch_clean": 0, 2024-09-13T12:17:07.549 INFO:tasks.workunit.client.0.smithi032.stdout: "source_version": "0'0", 2024-09-13T12:17:07.549 INFO:tasks.workunit.client.0.smithi032.stdout: "target_version": "0'0" 2024-09-13T12:17:07.549 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.549 INFO:tasks.workunit.client.0.smithi032.stdout: "last_change": "30", 2024-09-13T12:17:07.549 INFO:tasks.workunit.client.0.smithi032.stdout: "last_force_op_resend": "0", 2024-09-13T12:17:07.549 INFO:tasks.workunit.client.0.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T12:17:07.549 INFO:tasks.workunit.client.0.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T12:17:07.550 INFO:tasks.workunit.client.0.smithi032.stdout: "auid": 0, 2024-09-13T12:17:07.550 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_mode": "pool", 2024-09-13T12:17:07.550 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_seq": 1, 2024-09-13T12:17:07.550 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_epoch": 30, 2024-09-13T12:17:07.550 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_snaps": [ 2024-09-13T12:17:07.550 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.550 INFO:tasks.workunit.client.0.smithi032.stdout: "snapid": 1, 2024-09-13T12:17:07.550 INFO:tasks.workunit.client.0.smithi032.stdout: "stamp": "2024-09-13T12:15:09.622310+0000", 2024-09-13T12:17:07.550 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "snap1" 2024-09-13T12:17:07.550 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.550 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:17:07.550 INFO:tasks.workunit.client.0.smithi032.stdout: "removed_snaps": "[]", 2024-09-13T12:17:07.550 INFO:tasks.workunit.client.0.smithi032.stdout: "quota_max_bytes": 0, 2024-09-13T12:17:07.551 INFO:tasks.workunit.client.0.smithi032.stdout: "quota_max_objects": 0, 2024-09-13T12:17:07.551 INFO:tasks.workunit.client.0.smithi032.stdout: "tiers": [], 2024-09-13T12:17:07.551 INFO:tasks.workunit.client.0.smithi032.stdout: "tier_of": -1, 2024-09-13T12:17:07.551 INFO:tasks.workunit.client.0.smithi032.stdout: "read_tier": -1, 2024-09-13T12:17:07.551 INFO:tasks.workunit.client.0.smithi032.stdout: "write_tier": -1, 2024-09-13T12:17:07.551 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_mode": "none", 2024-09-13T12:17:07.551 INFO:tasks.workunit.client.0.smithi032.stdout: "target_max_bytes": 0, 2024-09-13T12:17:07.551 INFO:tasks.workunit.client.0.smithi032.stdout: "target_max_objects": 0, 2024-09-13T12:17:07.551 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T12:17:07.551 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T12:17:07.551 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T12:17:07.551 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-13T12:17:07.551 INFO:tasks.workunit.client.0.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-13T12:17:07.552 INFO:tasks.workunit.client.0.smithi032.stdout: "erasure_code_profile": "myprofile", 2024-09-13T12:17:07.552 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_params": { 2024-09-13T12:17:07.552 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "none" 2024-09-13T12:17:07.552 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.552 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_period": 0, 2024-09-13T12:17:07.552 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_count": 0, 2024-09-13T12:17:07.552 INFO:tasks.workunit.client.0.smithi032.stdout: "use_gmt_hitset": true, 2024-09-13T12:17:07.552 INFO:tasks.workunit.client.0.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-13T12:17:07.552 INFO:tasks.workunit.client.0.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-13T12:17:07.552 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T12:17:07.552 INFO:tasks.workunit.client.0.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-13T12:17:07.552 INFO:tasks.workunit.client.0.smithi032.stdout: "grade_table": [], 2024-09-13T12:17:07.552 INFO:tasks.workunit.client.0.smithi032.stdout: "stripe_width": 8192, 2024-09-13T12:17:07.552 INFO:tasks.workunit.client.0.smithi032.stdout: "expected_num_objects": 0, 2024-09-13T12:17:07.553 INFO:tasks.workunit.client.0.smithi032.stdout: "fast_read": false, 2024-09-13T12:17:07.553 INFO:tasks.workunit.client.0.smithi032.stdout: "options": {}, 2024-09-13T12:17:07.553 INFO:tasks.workunit.client.0.smithi032.stdout: "application_metadata": {} 2024-09-13T12:17:07.553 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.553 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:17:07.553 INFO:tasks.workunit.client.0.smithi032.stdout: "osds": [ 2024-09-13T12:17:07.553 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.553 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 0, 2024-09-13T12:17:07.553 INFO:tasks.workunit.client.0.smithi032.stdout: "uuid": "92bce19c-5a5e-4360-a088-1bf366fc2ecf", 2024-09-13T12:17:07.553 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 1, 2024-09-13T12:17:07.553 INFO:tasks.workunit.client.0.smithi032.stdout: "in": 1, 2024-09-13T12:17:07.553 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 1, 2024-09-13T12:17:07.553 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_affinity": 1, 2024-09-13T12:17:07.553 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_begin": 37, 2024-09-13T12:17:07.554 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_end": 42, 2024-09-13T12:17:07.554 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 45, 2024-09-13T12:17:07.554 INFO:tasks.workunit.client.0.smithi032.stdout: "up_thru": 49, 2024-09-13T12:17:07.554 INFO:tasks.workunit.client.0.smithi032.stdout: "down_at": 44, 2024-09-13T12:17:07.554 INFO:tasks.workunit.client.0.smithi032.stdout: "lost_at": 0, 2024-09-13T12:17:07.554 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addrs": { 2024-09-13T12:17:07.554 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:17:07.554 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.554 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:17:07.554 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6802", 2024-09-13T12:17:07.554 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 387212495 2024-09-13T12:17:07.554 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.554 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.555 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:17:07.555 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6803", 2024-09-13T12:17:07.555 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 387212495 2024-09-13T12:17:07.555 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.555 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:17:07.555 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.555 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addrs": { 2024-09-13T12:17:07.555 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:17:07.555 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.555 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:17:07.555 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6804", 2024-09-13T12:17:07.555 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 387212495 2024-09-13T12:17:07.555 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.555 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.556 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:17:07.556 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6805", 2024-09-13T12:17:07.556 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 387212495 2024-09-13T12:17:07.556 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.556 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:17:07.556 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.556 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-13T12:17:07.556 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:17:07.556 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.556 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:17:07.556 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6808", 2024-09-13T12:17:07.556 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 387212495 2024-09-13T12:17:07.556 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.556 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.557 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:17:07.557 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6809", 2024-09-13T12:17:07.557 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 387212495 2024-09-13T12:17:07.557 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.557 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:17:07.557 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.557 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-13T12:17:07.557 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:17:07.557 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.557 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:17:07.557 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6806", 2024-09-13T12:17:07.557 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 387212495 2024-09-13T12:17:07.557 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.557 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.558 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:17:07.558 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6807", 2024-09-13T12:17:07.558 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 387212495 2024-09-13T12:17:07.558 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.558 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:17:07.558 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.558 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addr": "127.0.0.1:6803/387212495", 2024-09-13T12:17:07.558 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addr": "127.0.0.1:6805/387212495", 2024-09-13T12:17:07.558 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addr": "127.0.0.1:6809/387212495", 2024-09-13T12:17:07.558 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addr": "127.0.0.1:6807/387212495", 2024-09-13T12:17:07.558 INFO:tasks.workunit.client.0.smithi032.stdout: "state": [ 2024-09-13T12:17:07.558 INFO:tasks.workunit.client.0.smithi032.stdout: "exists", 2024-09-13T12:17:07.558 INFO:tasks.workunit.client.0.smithi032.stdout: "up" 2024-09-13T12:17:07.559 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:17:07.559 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.559 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.559 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 1, 2024-09-13T12:17:07.559 INFO:tasks.workunit.client.0.smithi032.stdout: "uuid": "b6fb1428-41f7-46eb-8099-15ef8f53355b", 2024-09-13T12:17:07.559 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 0, 2024-09-13T12:17:07.559 INFO:tasks.workunit.client.0.smithi032.stdout: "in": 1, 2024-09-13T12:17:07.559 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 1, 2024-09-13T12:17:07.559 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_affinity": 1, 2024-09-13T12:17:07.559 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_begin": 0, 2024-09-13T12:17:07.559 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_end": 0, 2024-09-13T12:17:07.559 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 10, 2024-09-13T12:17:07.559 INFO:tasks.workunit.client.0.smithi032.stdout: "up_thru": 25, 2024-09-13T12:17:07.560 INFO:tasks.workunit.client.0.smithi032.stdout: "down_at": 32, 2024-09-13T12:17:07.560 INFO:tasks.workunit.client.0.smithi032.stdout: "lost_at": 0, 2024-09-13T12:17:07.560 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addrs": { 2024-09-13T12:17:07.560 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:17:07.560 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.560 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:17:07.560 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6810", 2024-09-13T12:17:07.560 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3755373257 2024-09-13T12:17:07.560 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.560 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.560 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:17:07.560 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6811", 2024-09-13T12:17:07.560 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3755373257 2024-09-13T12:17:07.561 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.561 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:17:07.561 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.561 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addrs": { 2024-09-13T12:17:07.561 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:17:07.561 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.561 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:17:07.561 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6812", 2024-09-13T12:17:07.561 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3755373257 2024-09-13T12:17:07.561 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.561 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.561 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:17:07.561 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6813", 2024-09-13T12:17:07.561 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3755373257 2024-09-13T12:17:07.562 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.562 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:17:07.562 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.562 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-13T12:17:07.562 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:17:07.562 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.562 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:17:07.562 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6816", 2024-09-13T12:17:07.562 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3755373257 2024-09-13T12:17:07.562 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.562 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.562 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:17:07.562 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6817", 2024-09-13T12:17:07.562 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3755373257 2024-09-13T12:17:07.563 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.563 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:17:07.563 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.563 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-13T12:17:07.563 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:17:07.563 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.563 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:17:07.563 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6814", 2024-09-13T12:17:07.563 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3755373257 2024-09-13T12:17:07.563 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.563 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.563 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:17:07.563 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6815", 2024-09-13T12:17:07.563 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 3755373257 2024-09-13T12:17:07.563 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.564 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:17:07.564 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.564 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addr": "127.0.0.1:6811/3755373257", 2024-09-13T12:17:07.564 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addr": "127.0.0.1:6813/3755373257", 2024-09-13T12:17:07.564 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3755373257", 2024-09-13T12:17:07.564 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3755373257", 2024-09-13T12:17:07.564 INFO:tasks.workunit.client.0.smithi032.stdout: "state": [ 2024-09-13T12:17:07.564 INFO:tasks.workunit.client.0.smithi032.stdout: "exists" 2024-09-13T12:17:07.564 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:17:07.564 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.564 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.564 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2, 2024-09-13T12:17:07.564 INFO:tasks.workunit.client.0.smithi032.stdout: "uuid": "575c91a2-dd41-4141-bbec-f3711b8a1a2e", 2024-09-13T12:17:07.565 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 1, 2024-09-13T12:17:07.565 INFO:tasks.workunit.client.0.smithi032.stdout: "in": 1, 2024-09-13T12:17:07.565 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 1, 2024-09-13T12:17:07.565 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_affinity": 1, 2024-09-13T12:17:07.565 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_begin": 39, 2024-09-13T12:17:07.565 INFO:tasks.workunit.client.0.smithi032.stdout: "last_clean_end": 42, 2024-09-13T12:17:07.565 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 49, 2024-09-13T12:17:07.565 INFO:tasks.workunit.client.0.smithi032.stdout: "up_thru": 0, 2024-09-13T12:17:07.565 INFO:tasks.workunit.client.0.smithi032.stdout: "down_at": 48, 2024-09-13T12:17:07.565 INFO:tasks.workunit.client.0.smithi032.stdout: "lost_at": 0, 2024-09-13T12:17:07.565 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addrs": { 2024-09-13T12:17:07.565 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:17:07.565 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.565 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:17:07.566 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6810", 2024-09-13T12:17:07.566 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 268464394 2024-09-13T12:17:07.566 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.566 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.566 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:17:07.566 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6811", 2024-09-13T12:17:07.566 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 268464394 2024-09-13T12:17:07.566 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.566 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:17:07.566 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.566 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addrs": { 2024-09-13T12:17:07.566 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:17:07.566 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.566 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:17:07.566 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6812", 2024-09-13T12:17:07.567 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 268464394 2024-09-13T12:17:07.567 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.567 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.567 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:17:07.567 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6813", 2024-09-13T12:17:07.567 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 268464394 2024-09-13T12:17:07.567 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.567 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:17:07.567 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.567 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-13T12:17:07.567 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:17:07.567 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.567 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:17:07.567 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6816", 2024-09-13T12:17:07.567 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 268464394 2024-09-13T12:17:07.568 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.568 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.568 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:17:07.568 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6817", 2024-09-13T12:17:07.568 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 268464394 2024-09-13T12:17:07.568 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.568 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:17:07.568 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.568 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-13T12:17:07.568 INFO:tasks.workunit.client.0.smithi032.stdout: "addrvec": [ 2024-09-13T12:17:07.568 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.568 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v2", 2024-09-13T12:17:07.568 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6814", 2024-09-13T12:17:07.568 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 268464394 2024-09-13T12:17:07.569 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.569 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.569 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "v1", 2024-09-13T12:17:07.569 INFO:tasks.workunit.client.0.smithi032.stdout: "addr": "127.0.0.1:6815", 2024-09-13T12:17:07.569 INFO:tasks.workunit.client.0.smithi032.stdout: "nonce": 268464394 2024-09-13T12:17:07.569 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.569 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:17:07.569 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.569 INFO:tasks.workunit.client.0.smithi032.stdout: "public_addr": "127.0.0.1:6811/268464394", 2024-09-13T12:17:07.569 INFO:tasks.workunit.client.0.smithi032.stdout: "cluster_addr": "127.0.0.1:6813/268464394", 2024-09-13T12:17:07.569 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_back_addr": "127.0.0.1:6817/268464394", 2024-09-13T12:17:07.569 INFO:tasks.workunit.client.0.smithi032.stdout: "heartbeat_front_addr": "127.0.0.1:6815/268464394", 2024-09-13T12:17:07.569 INFO:tasks.workunit.client.0.smithi032.stdout: "state": [ 2024-09-13T12:17:07.569 INFO:tasks.workunit.client.0.smithi032.stdout: "exists", 2024-09-13T12:17:07.570 INFO:tasks.workunit.client.0.smithi032.stdout: "up" 2024-09-13T12:17:07.570 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:17:07.570 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.570 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:17:07.570 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_xinfo": [ 2024-09-13T12:17:07.570 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.570 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 0, 2024-09-13T12:17:07.570 INFO:tasks.workunit.client.0.smithi032.stdout: "down_stamp": "2024-09-13T12:16:24.381395+0000", 2024-09-13T12:17:07.570 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_probability": 0, 2024-09-13T12:17:07.570 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_interval": 0, 2024-09-13T12:17:07.570 INFO:tasks.workunit.client.0.smithi032.stdout: "features": 4540719139924082687, 2024-09-13T12:17:07.570 INFO:tasks.workunit.client.0.smithi032.stdout: "old_weight": 0, 2024-09-13T12:17:07.570 INFO:tasks.workunit.client.0.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-13T12:13:04.824763+0000", 2024-09-13T12:17:07.570 INFO:tasks.workunit.client.0.smithi032.stdout: "dead_epoch": 0 2024-09-13T12:17:07.571 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.571 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.571 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 1, 2024-09-13T12:17:07.571 INFO:tasks.workunit.client.0.smithi032.stdout: "down_stamp": "2024-09-13T12:15:21.265408+0000", 2024-09-13T12:17:07.571 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_probability": 0, 2024-09-13T12:17:07.571 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_interval": 0, 2024-09-13T12:17:07.571 INFO:tasks.workunit.client.0.smithi032.stdout: "features": 4540719139924082687, 2024-09-13T12:17:07.571 INFO:tasks.workunit.client.0.smithi032.stdout: "old_weight": 0, 2024-09-13T12:17:07.571 INFO:tasks.workunit.client.0.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-13T12:13:41.500292+0000", 2024-09-13T12:17:07.571 INFO:tasks.workunit.client.0.smithi032.stdout: "dead_epoch": 0 2024-09-13T12:17:07.571 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.571 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.571 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2, 2024-09-13T12:17:07.572 INFO:tasks.workunit.client.0.smithi032.stdout: "down_stamp": "2024-09-13T12:16:27.354319+0000", 2024-09-13T12:17:07.572 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_probability": 0, 2024-09-13T12:17:07.572 INFO:tasks.workunit.client.0.smithi032.stdout: "laggy_interval": 0, 2024-09-13T12:17:07.572 INFO:tasks.workunit.client.0.smithi032.stdout: "features": 4540719139924082687, 2024-09-13T12:17:07.572 INFO:tasks.workunit.client.0.smithi032.stdout: "old_weight": 0, 2024-09-13T12:17:07.572 INFO:tasks.workunit.client.0.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-13T12:14:17.639955+0000", 2024-09-13T12:17:07.572 INFO:tasks.workunit.client.0.smithi032.stdout: "dead_epoch": 0 2024-09-13T12:17:07.572 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.572 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:17:07.572 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_upmap": [], 2024-09-13T12:17:07.572 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_upmap_items": [], 2024-09-13T12:17:07.572 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_upmap_primaries": [], 2024-09-13T12:17:07.572 INFO:tasks.workunit.client.0.smithi032.stdout: "pg_temp": [], 2024-09-13T12:17:07.572 INFO:tasks.workunit.client.0.smithi032.stdout: "primary_temp": [], 2024-09-13T12:17:07.573 INFO:tasks.workunit.client.0.smithi032.stdout: "blocklist": {}, 2024-09-13T12:17:07.573 INFO:tasks.workunit.client.0.smithi032.stdout: "range_blocklist": {}, 2024-09-13T12:17:07.573 INFO:tasks.workunit.client.0.smithi032.stdout: "erasure_code_profiles": { 2024-09-13T12:17:07.573 INFO:tasks.workunit.client.0.smithi032.stdout: "default": { 2024-09-13T12:17:07.573 INFO:tasks.workunit.client.0.smithi032.stdout: "k": "2", 2024-09-13T12:17:07.573 INFO:tasks.workunit.client.0.smithi032.stdout: "m": "2", 2024-09-13T12:17:07.573 INFO:tasks.workunit.client.0.smithi032.stdout: "plugin": "jerasure", 2024-09-13T12:17:07.573 INFO:tasks.workunit.client.0.smithi032.stdout: "technique": "reed_sol_van" 2024-09-13T12:17:07.573 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.573 INFO:tasks.workunit.client.0.smithi032.stdout: "myprofile": { 2024-09-13T12:17:07.573 INFO:tasks.workunit.client.0.smithi032.stdout: "crush-device-class": "", 2024-09-13T12:17:07.573 INFO:tasks.workunit.client.0.smithi032.stdout: "crush-failure-domain": "osd", 2024-09-13T12:17:07.573 INFO:tasks.workunit.client.0.smithi032.stdout: "crush-num-failure-domains": "0", 2024-09-13T12:17:07.573 INFO:tasks.workunit.client.0.smithi032.stdout: "crush-osds-per-failure-domain": "0", 2024-09-13T12:17:07.573 INFO:tasks.workunit.client.0.smithi032.stdout: "crush-root": "default", 2024-09-13T12:17:07.574 INFO:tasks.workunit.client.0.smithi032.stdout: "jerasure-per-chunk-alignment": "false", 2024-09-13T12:17:07.574 INFO:tasks.workunit.client.0.smithi032.stdout: "k": "2", 2024-09-13T12:17:07.574 INFO:tasks.workunit.client.0.smithi032.stdout: "m": "1", 2024-09-13T12:17:07.574 INFO:tasks.workunit.client.0.smithi032.stdout: "plugin": "jerasure", 2024-09-13T12:17:07.574 INFO:tasks.workunit.client.0.smithi032.stdout: "technique": "reed_sol_van", 2024-09-13T12:17:07.574 INFO:tasks.workunit.client.0.smithi032.stdout: "w": "8" 2024-09-13T12:17:07.574 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.574 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.574 INFO:tasks.workunit.client.0.smithi032.stdout: "removed_snaps_queue": [], 2024-09-13T12:17:07.574 INFO:tasks.workunit.client.0.smithi032.stdout: "new_removed_snaps": [], 2024-09-13T12:17:07.574 INFO:tasks.workunit.client.0.smithi032.stdout: "new_purged_snaps": [], 2024-09-13T12:17:07.574 INFO:tasks.workunit.client.0.smithi032.stdout: "crush_node_flags": {}, 2024-09-13T12:17:07.574 INFO:tasks.workunit.client.0.smithi032.stdout: "device_class_flags": {}, 2024-09-13T12:17:07.574 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_mode": { 2024-09-13T12:17:07.575 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_mode_enabled": false, 2024-09-13T12:17:07.575 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_bucket_count": 0, 2024-09-13T12:17:07.575 INFO:tasks.workunit.client.0.smithi032.stdout: "degraded_stretch_mode": 0, 2024-09-13T12:17:07.575 INFO:tasks.workunit.client.0.smithi032.stdout: "recovering_stretch_mode": 0, 2024-09-13T12:17:07.575 INFO:tasks.workunit.client.0.smithi032.stdout: "stretch_mode_bucket": 0 2024-09-13T12:17:07.575 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.575 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.575 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_metadata": [ 2024-09-13T12:17:07.575 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.575 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:17:07.576 INFO:tasks.workunit.client.0.smithi032.stdout: "arch": "x86_64", 2024-09-13T12:17:07.576 INFO:tasks.workunit.client.0.smithi032.stdout: "back_addr": "[v2:127.0.0.1:6804/387212495,v1:127.0.0.1:6805/387212495]", 2024-09-13T12:17:07.576 INFO:tasks.workunit.client.0.smithi032.stdout: "back_iface": "", 2024-09-13T12:17:07.576 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs": "1", 2024-09-13T12:17:07.576 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_db": "0", 2024-09-13T12:17:07.576 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_wal": "0", 2024-09-13T12:17:07.576 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_single_shared_device": "1", 2024-09-13T12:17:07.576 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_allocation_from_file": "0", 2024-09-13T12:17:07.576 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_access_mode": "file", 2024-09-13T12:17:07.576 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_block_size": "4096", 2024-09-13T12:17:07.576 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_devices": "sda", 2024-09-13T12:17:07.576 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-13T12:17:07.576 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-13T12:17:07.576 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-13T12:17:07.577 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_rotational": "1", 2024-09-13T12:17:07.577 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_size": "107374182400", 2024-09-13T12:17:07.577 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_support_discard": "1", 2024-09-13T12:17:07.577 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_type": "hdd", 2024-09-13T12:17:07.577 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_min_alloc_size": "4096", 2024-09-13T12:17:07.577 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_release": "squid", 2024-09-13T12:17:07.577 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:17:07.577 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_short": "19.3.0-4697-gd58f7306", 2024-09-13T12:17:07.577 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_when_created": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:17:07.577 INFO:tasks.workunit.client.0.smithi032.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-13T12:17:07.577 INFO:tasks.workunit.client.0.smithi032.stdout: "created_at": "2024-09-13T12:12:37.432453Z", 2024-09-13T12:17:07.577 INFO:tasks.workunit.client.0.smithi032.stdout: "default_device_class": "hdd", 2024-09-13T12:17:07.577 INFO:tasks.workunit.client.0.smithi032.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4J2LL", 2024-09-13T12:17:07.577 INFO:tasks.workunit.client.0.smithi032.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-13T12:17:07.578 INFO:tasks.workunit.client.0.smithi032.stdout: "devices": "sda", 2024-09-13T12:17:07.578 INFO:tasks.workunit.client.0.smithi032.stdout: "distro": "centos", 2024-09-13T12:17:07.578 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_description": "CentOS Stream 9", 2024-09-13T12:17:07.578 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_version": "9", 2024-09-13T12:17:07.578 INFO:tasks.workunit.client.0.smithi032.stdout: "front_addr": "[v2:127.0.0.1:6802/387212495,v1:127.0.0.1:6803/387212495]", 2024-09-13T12:17:07.578 INFO:tasks.workunit.client.0.smithi032.stdout: "front_iface": "", 2024-09-13T12:17:07.578 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/387212495,v1:127.0.0.1:6809/387212495]", 2024-09-13T12:17:07.578 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/387212495,v1:127.0.0.1:6807/387212495]", 2024-09-13T12:17:07.578 INFO:tasks.workunit.client.0.smithi032.stdout: "hostname": "smithi032", 2024-09-13T12:17:07.578 INFO:tasks.workunit.client.0.smithi032.stdout: "journal_rotational": "1", 2024-09-13T12:17:07.578 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-13T12:17:07.578 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-13T12:17:07.578 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_swap_kb": "0", 2024-09-13T12:17:07.579 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_total_kb": "32493484", 2024-09-13T12:17:07.579 INFO:tasks.workunit.client.0.smithi032.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-13T12:17:07.579 INFO:tasks.workunit.client.0.smithi032.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-13T12:17:07.579 INFO:tasks.workunit.client.0.smithi032.stdout: "os": "Linux", 2024-09-13T12:17:07.579 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_data": "td/divergent-priors/0", 2024-09-13T12:17:07.579 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_objectstore": "bluestore", 2024-09-13T12:17:07.579 INFO:tasks.workunit.client.0.smithi032.stdout: "osdspec_affinity": "", 2024-09-13T12:17:07.579 INFO:tasks.workunit.client.0.smithi032.stdout: "rotational": "1" 2024-09-13T12:17:07.579 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.579 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.579 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 1, 2024-09-13T12:17:07.579 INFO:tasks.workunit.client.0.smithi032.stdout: "arch": "x86_64", 2024-09-13T12:17:07.579 INFO:tasks.workunit.client.0.smithi032.stdout: "back_addr": "[v2:127.0.0.1:6812/3755373257,v1:127.0.0.1:6813/3755373257]", 2024-09-13T12:17:07.579 INFO:tasks.workunit.client.0.smithi032.stdout: "back_iface": "", 2024-09-13T12:17:07.580 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs": "1", 2024-09-13T12:17:07.580 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_db": "0", 2024-09-13T12:17:07.580 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_wal": "0", 2024-09-13T12:17:07.580 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_single_shared_device": "1", 2024-09-13T12:17:07.580 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_allocation_from_file": "0", 2024-09-13T12:17:07.580 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_access_mode": "file", 2024-09-13T12:17:07.580 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_block_size": "4096", 2024-09-13T12:17:07.580 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_devices": "sda", 2024-09-13T12:17:07.580 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-13T12:17:07.580 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-13T12:17:07.580 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-13T12:17:07.580 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_rotational": "1", 2024-09-13T12:17:07.580 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_size": "107374182400", 2024-09-13T12:17:07.580 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_support_discard": "1", 2024-09-13T12:17:07.581 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_type": "hdd", 2024-09-13T12:17:07.581 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_min_alloc_size": "4096", 2024-09-13T12:17:07.581 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_release": "squid", 2024-09-13T12:17:07.581 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:17:07.581 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_short": "19.3.0-4697-gd58f7306", 2024-09-13T12:17:07.581 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_when_created": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:17:07.581 INFO:tasks.workunit.client.0.smithi032.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-13T12:17:07.581 INFO:tasks.workunit.client.0.smithi032.stdout: "created_at": "2024-09-13T12:13:12.366247Z", 2024-09-13T12:17:07.581 INFO:tasks.workunit.client.0.smithi032.stdout: "default_device_class": "hdd", 2024-09-13T12:17:07.581 INFO:tasks.workunit.client.0.smithi032.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4J2LL", 2024-09-13T12:17:07.581 INFO:tasks.workunit.client.0.smithi032.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-13T12:17:07.581 INFO:tasks.workunit.client.0.smithi032.stdout: "devices": "sda", 2024-09-13T12:17:07.581 INFO:tasks.workunit.client.0.smithi032.stdout: "distro": "centos", 2024-09-13T12:17:07.581 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_description": "CentOS Stream 9", 2024-09-13T12:17:07.581 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_version": "9", 2024-09-13T12:17:07.582 INFO:tasks.workunit.client.0.smithi032.stdout: "front_addr": "[v2:127.0.0.1:6810/3755373257,v1:127.0.0.1:6811/3755373257]", 2024-09-13T12:17:07.582 INFO:tasks.workunit.client.0.smithi032.stdout: "front_iface": "", 2024-09-13T12:17:07.582 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3755373257,v1:127.0.0.1:6817/3755373257]", 2024-09-13T12:17:07.582 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3755373257,v1:127.0.0.1:6815/3755373257]", 2024-09-13T12:17:07.582 INFO:tasks.workunit.client.0.smithi032.stdout: "hostname": "smithi032", 2024-09-13T12:17:07.582 INFO:tasks.workunit.client.0.smithi032.stdout: "journal_rotational": "1", 2024-09-13T12:17:07.582 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-13T12:17:07.582 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-13T12:17:07.582 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_swap_kb": "0", 2024-09-13T12:17:07.582 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_total_kb": "32493484", 2024-09-13T12:17:07.582 INFO:tasks.workunit.client.0.smithi032.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-13T12:17:07.582 INFO:tasks.workunit.client.0.smithi032.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-13T12:17:07.582 INFO:tasks.workunit.client.0.smithi032.stdout: "os": "Linux", 2024-09-13T12:17:07.582 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_data": "td/divergent-priors/1", 2024-09-13T12:17:07.583 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_objectstore": "bluestore", 2024-09-13T12:17:07.583 INFO:tasks.workunit.client.0.smithi032.stdout: "osdspec_affinity": "", 2024-09-13T12:17:07.583 INFO:tasks.workunit.client.0.smithi032.stdout: "rotational": "1" 2024-09-13T12:17:07.583 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.583 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.583 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:17:07.583 INFO:tasks.workunit.client.0.smithi032.stdout: "arch": "x86_64", 2024-09-13T12:17:07.583 INFO:tasks.workunit.client.0.smithi032.stdout: "back_addr": "[v2:127.0.0.1:6812/268464394,v1:127.0.0.1:6813/268464394]", 2024-09-13T12:17:07.583 INFO:tasks.workunit.client.0.smithi032.stdout: "back_iface": "", 2024-09-13T12:17:07.583 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs": "1", 2024-09-13T12:17:07.584 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_db": "0", 2024-09-13T12:17:07.584 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_dedicated_wal": "0", 2024-09-13T12:17:07.584 INFO:tasks.workunit.client.0.smithi032.stdout: "bluefs_single_shared_device": "1", 2024-09-13T12:17:07.584 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_allocation_from_file": "0", 2024-09-13T12:17:07.584 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_access_mode": "file", 2024-09-13T12:17:07.584 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_block_size": "4096", 2024-09-13T12:17:07.584 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_devices": "sda", 2024-09-13T12:17:07.584 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-13T12:17:07.584 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-13T12:17:07.584 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-13T12:17:07.584 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_rotational": "1", 2024-09-13T12:17:07.584 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_size": "107374182400", 2024-09-13T12:17:07.585 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_support_discard": "1", 2024-09-13T12:17:07.585 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_bdev_type": "hdd", 2024-09-13T12:17:07.585 INFO:tasks.workunit.client.0.smithi032.stdout: "bluestore_min_alloc_size": "4096", 2024-09-13T12:17:07.585 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_release": "squid", 2024-09-13T12:17:07.585 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:17:07.585 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_short": "19.3.0-4697-gd58f7306", 2024-09-13T12:17:07.585 INFO:tasks.workunit.client.0.smithi032.stdout: "ceph_version_when_created": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T12:17:07.585 INFO:tasks.workunit.client.0.smithi032.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-13T12:17:07.585 INFO:tasks.workunit.client.0.smithi032.stdout: "created_at": "2024-09-13T12:13:49.134953Z", 2024-09-13T12:17:07.585 INFO:tasks.workunit.client.0.smithi032.stdout: "default_device_class": "hdd", 2024-09-13T12:17:07.585 INFO:tasks.workunit.client.0.smithi032.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4J2LL", 2024-09-13T12:17:07.585 INFO:tasks.workunit.client.0.smithi032.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-13T12:17:07.585 INFO:tasks.workunit.client.0.smithi032.stdout: "devices": "sda", 2024-09-13T12:17:07.586 INFO:tasks.workunit.client.0.smithi032.stdout: "distro": "centos", 2024-09-13T12:17:07.586 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_description": "CentOS Stream 9", 2024-09-13T12:17:07.586 INFO:tasks.workunit.client.0.smithi032.stdout: "distro_version": "9", 2024-09-13T12:17:07.586 INFO:tasks.workunit.client.0.smithi032.stdout: "front_addr": "[v2:127.0.0.1:6810/268464394,v1:127.0.0.1:6811/268464394]", 2024-09-13T12:17:07.586 INFO:tasks.workunit.client.0.smithi032.stdout: "front_iface": "", 2024-09-13T12:17:07.586 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/268464394,v1:127.0.0.1:6817/268464394]", 2024-09-13T12:17:07.586 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/268464394,v1:127.0.0.1:6815/268464394]", 2024-09-13T12:17:07.586 INFO:tasks.workunit.client.0.smithi032.stdout: "hostname": "smithi032", 2024-09-13T12:17:07.586 INFO:tasks.workunit.client.0.smithi032.stdout: "journal_rotational": "1", 2024-09-13T12:17:07.586 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-13T12:17:07.586 INFO:tasks.workunit.client.0.smithi032.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-13T12:17:07.586 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_swap_kb": "0", 2024-09-13T12:17:07.586 INFO:tasks.workunit.client.0.smithi032.stdout: "mem_total_kb": "32493484", 2024-09-13T12:17:07.586 INFO:tasks.workunit.client.0.smithi032.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-13T12:17:07.587 INFO:tasks.workunit.client.0.smithi032.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-13T12:17:07.587 INFO:tasks.workunit.client.0.smithi032.stdout: "os": "Linux", 2024-09-13T12:17:07.587 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_data": "td/divergent-priors/2", 2024-09-13T12:17:07.587 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_objectstore": "bluestore", 2024-09-13T12:17:07.587 INFO:tasks.workunit.client.0.smithi032.stdout: "osdspec_affinity": "", 2024-09-13T12:17:07.587 INFO:tasks.workunit.client.0.smithi032.stdout: "rotational": "1" 2024-09-13T12:17:07.587 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.587 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:17:07.587 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap_clean_epochs": { 2024-09-13T12:17:07.587 INFO:tasks.workunit.client.0.smithi032.stdout: "min_last_epoch_clean": 27, 2024-09-13T12:17:07.587 INFO:tasks.workunit.client.0.smithi032.stdout: "last_epoch_clean": { 2024-09-13T12:17:07.587 INFO:tasks.workunit.client.0.smithi032.stdout: "per_pool": [ 2024-09-13T12:17:07.587 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.587 INFO:tasks.workunit.client.0.smithi032.stdout: "poolid": 1, 2024-09-13T12:17:07.588 INFO:tasks.workunit.client.0.smithi032.stdout: "floor": 27 2024-09-13T12:17:07.588 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.588 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:17:07.588 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.588 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_epochs": [ 2024-09-13T12:17:07.588 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.588 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:17:07.588 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 45 2024-09-13T12:17:07.588 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.588 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.588 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:17:07.588 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 49 2024-09-13T12:17:07.588 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.589 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:17:07.589 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.589 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap_first_committed": 1, 2024-09-13T12:17:07.589 INFO:tasks.workunit.client.0.smithi032.stdout: "osdmap_last_committed": 50, 2024-09-13T12:17:07.589 INFO:tasks.workunit.client.0.smithi032.stdout: "crushmap": { 2024-09-13T12:17:07.589 INFO:tasks.workunit.client.0.smithi032.stdout: "devices": [ 2024-09-13T12:17:07.589 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.589 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:17:07.589 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "osd.0", 2024-09-13T12:17:07.589 INFO:tasks.workunit.client.0.smithi032.stdout: "class": "hdd" 2024-09-13T12:17:07.589 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.589 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.589 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 1, 2024-09-13T12:17:07.590 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "osd.1", 2024-09-13T12:17:07.590 INFO:tasks.workunit.client.0.smithi032.stdout: "class": "hdd" 2024-09-13T12:17:07.590 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.590 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.590 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:17:07.590 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "osd.2", 2024-09-13T12:17:07.590 INFO:tasks.workunit.client.0.smithi032.stdout: "class": "hdd" 2024-09-13T12:17:07.590 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.590 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:17:07.590 INFO:tasks.workunit.client.0.smithi032.stdout: "types": [ 2024-09-13T12:17:07.590 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.590 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 0, 2024-09-13T12:17:07.590 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "osd" 2024-09-13T12:17:07.591 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.591 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.591 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 1, 2024-09-13T12:17:07.591 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "host" 2024-09-13T12:17:07.591 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.591 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.591 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 2, 2024-09-13T12:17:07.591 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "chassis" 2024-09-13T12:17:07.591 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.591 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.591 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 3, 2024-09-13T12:17:07.591 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "rack" 2024-09-13T12:17:07.591 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.591 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.592 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 4, 2024-09-13T12:17:07.592 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "row" 2024-09-13T12:17:07.592 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.592 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.592 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 5, 2024-09-13T12:17:07.592 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "pdu" 2024-09-13T12:17:07.592 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.592 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.592 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 6, 2024-09-13T12:17:07.592 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "pod" 2024-09-13T12:17:07.592 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.592 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.592 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 7, 2024-09-13T12:17:07.592 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "room" 2024-09-13T12:17:07.592 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.593 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.593 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 8, 2024-09-13T12:17:07.593 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "datacenter" 2024-09-13T12:17:07.593 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.593 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.593 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 9, 2024-09-13T12:17:07.593 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "zone" 2024-09-13T12:17:07.593 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.593 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.593 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 10, 2024-09-13T12:17:07.593 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "region" 2024-09-13T12:17:07.593 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.594 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.594 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 11, 2024-09-13T12:17:07.594 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "root" 2024-09-13T12:17:07.594 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.594 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:17:07.594 INFO:tasks.workunit.client.0.smithi032.stdout: "buckets": [ 2024-09-13T12:17:07.594 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.594 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -1, 2024-09-13T12:17:07.594 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "default", 2024-09-13T12:17:07.594 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 11, 2024-09-13T12:17:07.594 INFO:tasks.workunit.client.0.smithi032.stdout: "type_name": "root", 2024-09-13T12:17:07.594 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:17:07.595 INFO:tasks.workunit.client.0.smithi032.stdout: "alg": "straw2", 2024-09-13T12:17:07.595 INFO:tasks.workunit.client.0.smithi032.stdout: "hash": "rjenkins1", 2024-09-13T12:17:07.595 INFO:tasks.workunit.client.0.smithi032.stdout: "items": [ 2024-09-13T12:17:07.595 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.595 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -3, 2024-09-13T12:17:07.595 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:17:07.595 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 0 2024-09-13T12:17:07.595 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.595 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:17:07.595 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.595 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.595 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -2, 2024-09-13T12:17:07.595 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "default~hdd", 2024-09-13T12:17:07.595 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 11, 2024-09-13T12:17:07.596 INFO:tasks.workunit.client.0.smithi032.stdout: "type_name": "root", 2024-09-13T12:17:07.596 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:17:07.596 INFO:tasks.workunit.client.0.smithi032.stdout: "alg": "straw2", 2024-09-13T12:17:07.596 INFO:tasks.workunit.client.0.smithi032.stdout: "hash": "rjenkins1", 2024-09-13T12:17:07.596 INFO:tasks.workunit.client.0.smithi032.stdout: "items": [ 2024-09-13T12:17:07.596 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.596 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -4, 2024-09-13T12:17:07.596 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:17:07.596 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 0 2024-09-13T12:17:07.596 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.596 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:17:07.596 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.596 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.597 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -3, 2024-09-13T12:17:07.597 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "smithi032", 2024-09-13T12:17:07.597 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 1, 2024-09-13T12:17:07.597 INFO:tasks.workunit.client.0.smithi032.stdout: "type_name": "host", 2024-09-13T12:17:07.597 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:17:07.597 INFO:tasks.workunit.client.0.smithi032.stdout: "alg": "straw2", 2024-09-13T12:17:07.597 INFO:tasks.workunit.client.0.smithi032.stdout: "hash": "rjenkins1", 2024-09-13T12:17:07.597 INFO:tasks.workunit.client.0.smithi032.stdout: "items": [ 2024-09-13T12:17:07.597 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.597 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:17:07.597 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:17:07.597 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 0 2024-09-13T12:17:07.597 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.598 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.598 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 1, 2024-09-13T12:17:07.598 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:17:07.598 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 1 2024-09-13T12:17:07.598 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.598 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.598 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:17:07.598 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:17:07.598 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 2 2024-09-13T12:17:07.598 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.598 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:17:07.598 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.598 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.598 INFO:tasks.workunit.client.0.smithi032.stdout: "id": -4, 2024-09-13T12:17:07.599 INFO:tasks.workunit.client.0.smithi032.stdout: "name": "smithi032~hdd", 2024-09-13T12:17:07.599 INFO:tasks.workunit.client.0.smithi032.stdout: "type_id": 1, 2024-09-13T12:17:07.599 INFO:tasks.workunit.client.0.smithi032.stdout: "type_name": "host", 2024-09-13T12:17:07.599 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 19206, 2024-09-13T12:17:07.599 INFO:tasks.workunit.client.0.smithi032.stdout: "alg": "straw2", 2024-09-13T12:17:07.599 INFO:tasks.workunit.client.0.smithi032.stdout: "hash": "rjenkins1", 2024-09-13T12:17:07.599 INFO:tasks.workunit.client.0.smithi032.stdout: "items": [ 2024-09-13T12:17:07.599 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.599 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 0, 2024-09-13T12:17:07.599 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:17:07.599 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 0 2024-09-13T12:17:07.599 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.599 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.599 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 1, 2024-09-13T12:17:07.599 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:17:07.600 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 1 2024-09-13T12:17:07.600 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.600 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.600 INFO:tasks.workunit.client.0.smithi032.stdout: "id": 2, 2024-09-13T12:17:07.600 INFO:tasks.workunit.client.0.smithi032.stdout: "weight": 6402, 2024-09-13T12:17:07.600 INFO:tasks.workunit.client.0.smithi032.stdout: "pos": 2 2024-09-13T12:17:07.600 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.600 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:17:07.600 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.600 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:17:07.600 INFO:tasks.workunit.client.0.smithi032.stdout: "rules": [ 2024-09-13T12:17:07.600 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.600 INFO:tasks.workunit.client.0.smithi032.stdout: "rule_id": 0, 2024-09-13T12:17:07.600 INFO:tasks.workunit.client.0.smithi032.stdout: "rule_name": "replicated_rule", 2024-09-13T12:17:07.601 INFO:tasks.workunit.client.0.smithi032.stdout: "type": 1, 2024-09-13T12:17:07.601 INFO:tasks.workunit.client.0.smithi032.stdout: "steps": [ 2024-09-13T12:17:07.601 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.601 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "take", 2024-09-13T12:17:07.601 INFO:tasks.workunit.client.0.smithi032.stdout: "item": -1, 2024-09-13T12:17:07.601 INFO:tasks.workunit.client.0.smithi032.stdout: "item_name": "default" 2024-09-13T12:17:07.601 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.601 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.601 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "choose_firstn", 2024-09-13T12:17:07.601 INFO:tasks.workunit.client.0.smithi032.stdout: "num": 0, 2024-09-13T12:17:07.601 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "osd" 2024-09-13T12:17:07.601 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.602 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.602 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "emit" 2024-09-13T12:17:07.602 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.602 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:17:07.602 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.602 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.602 INFO:tasks.workunit.client.0.smithi032.stdout: "rule_id": 1, 2024-09-13T12:17:07.602 INFO:tasks.workunit.client.0.smithi032.stdout: "rule_name": "test", 2024-09-13T12:17:07.602 INFO:tasks.workunit.client.0.smithi032.stdout: "type": 3, 2024-09-13T12:17:07.602 INFO:tasks.workunit.client.0.smithi032.stdout: "steps": [ 2024-09-13T12:17:07.602 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.602 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "set_chooseleaf_tries", 2024-09-13T12:17:07.602 INFO:tasks.workunit.client.0.smithi032.stdout: "num": 5 2024-09-13T12:17:07.602 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.603 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.603 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "set_choose_tries", 2024-09-13T12:17:07.603 INFO:tasks.workunit.client.0.smithi032.stdout: "num": 100 2024-09-13T12:17:07.603 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.603 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.603 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "take", 2024-09-13T12:17:07.603 INFO:tasks.workunit.client.0.smithi032.stdout: "item": -1, 2024-09-13T12:17:07.603 INFO:tasks.workunit.client.0.smithi032.stdout: "item_name": "default" 2024-09-13T12:17:07.603 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.603 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.603 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "choose_indep", 2024-09-13T12:17:07.603 INFO:tasks.workunit.client.0.smithi032.stdout: "num": 0, 2024-09-13T12:17:07.603 INFO:tasks.workunit.client.0.smithi032.stdout: "type": "osd" 2024-09-13T12:17:07.603 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.603 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.604 INFO:tasks.workunit.client.0.smithi032.stderr:report 1687148924 2024-09-13T12:17:07.604 INFO:tasks.workunit.client.0.smithi032.stdout: "op": "emit" 2024-09-13T12:17:07.604 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.604 INFO:tasks.workunit.client.0.smithi032.stdout: ] 2024-09-13T12:17:07.605 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.605 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:17:07.605 INFO:tasks.workunit.client.0.smithi032.stdout: "tunables": { 2024-09-13T12:17:07.605 INFO:tasks.workunit.client.0.smithi032.stdout: "choose_local_tries": 0, 2024-09-13T12:17:07.605 INFO:tasks.workunit.client.0.smithi032.stdout: "choose_local_fallback_tries": 0, 2024-09-13T12:17:07.605 INFO:tasks.workunit.client.0.smithi032.stdout: "choose_total_tries": 50, 2024-09-13T12:17:07.605 INFO:tasks.workunit.client.0.smithi032.stdout: "chooseleaf_descend_once": 1, 2024-09-13T12:17:07.605 INFO:tasks.workunit.client.0.smithi032.stdout: "chooseleaf_vary_r": 1, 2024-09-13T12:17:07.605 INFO:tasks.workunit.client.0.smithi032.stdout: "chooseleaf_stable": 1, 2024-09-13T12:17:07.605 INFO:tasks.workunit.client.0.smithi032.stdout: "msr_descents": 100, 2024-09-13T12:17:07.605 INFO:tasks.workunit.client.0.smithi032.stdout: "msr_collision_tries": 100, 2024-09-13T12:17:07.605 INFO:tasks.workunit.client.0.smithi032.stdout: "straw_calc_version": 1, 2024-09-13T12:17:07.605 INFO:tasks.workunit.client.0.smithi032.stdout: "allowed_bucket_algs": 54, 2024-09-13T12:17:07.605 INFO:tasks.workunit.client.0.smithi032.stdout: "profile": "jewel", 2024-09-13T12:17:07.606 INFO:tasks.workunit.client.0.smithi032.stdout: "optimal_tunables": 1, 2024-09-13T12:17:07.606 INFO:tasks.workunit.client.0.smithi032.stdout: "legacy_tunables": 0, 2024-09-13T12:17:07.606 INFO:tasks.workunit.client.0.smithi032.stdout: "minimum_required_version": "jewel", 2024-09-13T12:17:07.606 INFO:tasks.workunit.client.0.smithi032.stdout: "require_feature_tunables": 1, 2024-09-13T12:17:07.606 INFO:tasks.workunit.client.0.smithi032.stdout: "require_feature_tunables2": 1, 2024-09-13T12:17:07.606 INFO:tasks.workunit.client.0.smithi032.stdout: "has_v2_rules": 1, 2024-09-13T12:17:07.606 INFO:tasks.workunit.client.0.smithi032.stdout: "require_feature_tunables3": 1, 2024-09-13T12:17:07.606 INFO:tasks.workunit.client.0.smithi032.stdout: "has_v3_rules": 0, 2024-09-13T12:17:07.606 INFO:tasks.workunit.client.0.smithi032.stdout: "has_v4_buckets": 1, 2024-09-13T12:17:07.606 INFO:tasks.workunit.client.0.smithi032.stdout: "require_feature_tunables5": 1, 2024-09-13T12:17:07.606 INFO:tasks.workunit.client.0.smithi032.stdout: "has_v5_rules": 0, 2024-09-13T12:17:07.606 INFO:tasks.workunit.client.0.smithi032.stdout: "has_msr_rules": 0 2024-09-13T12:17:07.606 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.607 INFO:tasks.workunit.client.0.smithi032.stdout: "choose_args": {} 2024-09-13T12:17:07.607 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.607 INFO:tasks.workunit.client.0.smithi032.stdout: "fsmap": { 2024-09-13T12:17:07.607 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 1, 2024-09-13T12:17:07.607 INFO:tasks.workunit.client.0.smithi032.stdout: "btime": "2024-09-13T12:12:31:260139+0000", 2024-09-13T12:17:07.607 INFO:tasks.workunit.client.0.smithi032.stdout: "default_fscid": -1, 2024-09-13T12:17:07.607 INFO:tasks.workunit.client.0.smithi032.stdout: "compat": { 2024-09-13T12:17:07.607 INFO:tasks.workunit.client.0.smithi032.stdout: "compat": {}, 2024-09-13T12:17:07.607 INFO:tasks.workunit.client.0.smithi032.stdout: "ro_compat": {}, 2024-09-13T12:17:07.607 INFO:tasks.workunit.client.0.smithi032.stdout: "incompat": { 2024-09-13T12:17:07.607 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_1": "base v0.20", 2024-09-13T12:17:07.607 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_2": "client writeable ranges", 2024-09-13T12:17:07.607 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_3": "default file layouts on dirs", 2024-09-13T12:17:07.607 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_4": "dir inode in separate object", 2024-09-13T12:17:07.608 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_5": "mds uses versioned encoding", 2024-09-13T12:17:07.608 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-13T12:17:07.608 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_8": "no anchor table", 2024-09-13T12:17:07.608 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_9": "file layout v2", 2024-09-13T12:17:07.608 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_10": "snaprealm v2", 2024-09-13T12:17:07.608 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_11": "minor log segments", 2024-09-13T12:17:07.608 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_12": "quiesce subvolumes" 2024-09-13T12:17:07.608 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.608 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.608 INFO:tasks.workunit.client.0.smithi032.stdout: "feature_flags": { 2024-09-13T12:17:07.608 INFO:tasks.workunit.client.0.smithi032.stdout: "enable_multiple": true, 2024-09-13T12:17:07.608 INFO:tasks.workunit.client.0.smithi032.stdout: "ever_enabled_multiple": true 2024-09-13T12:17:07.608 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.609 INFO:tasks.workunit.client.0.smithi032.stdout: "standbys": [], 2024-09-13T12:17:07.609 INFO:tasks.workunit.client.0.smithi032.stdout: "filesystems": [] 2024-09-13T12:17:07.609 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.609 INFO:tasks.workunit.client.0.smithi032.stdout: "mdsmap_first_committed": 1, 2024-09-13T12:17:07.609 INFO:tasks.workunit.client.0.smithi032.stdout: "mdsmap_last_committed": 1, 2024-09-13T12:17:07.609 INFO:tasks.workunit.client.0.smithi032.stdout: "auth": { 2024-09-13T12:17:07.609 INFO:tasks.workunit.client.0.smithi032.stdout: "first_committed": 1, 2024-09-13T12:17:07.609 INFO:tasks.workunit.client.0.smithi032.stdout: "last_committed": 5, 2024-09-13T12:17:07.609 INFO:tasks.workunit.client.0.smithi032.stdout: "num_secrets": 3 2024-09-13T12:17:07.609 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.609 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg": 1, 2024-09-13T12:17:07.609 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg_active": 1, 2024-09-13T12:17:07.609 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg_unknown": 0, 2024-09-13T12:17:07.609 INFO:tasks.workunit.client.0.smithi032.stdout: "num_osd": 3, 2024-09-13T12:17:07.610 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_sum": { 2024-09-13T12:17:07.610 INFO:tasks.workunit.client.0.smithi032.stdout: "stat_sum": { 2024-09-13T12:17:07.610 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes": 48454, 2024-09-13T12:17:07.610 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects": 101, 2024-09-13T12:17:07.610 INFO:tasks.workunit.client.0.smithi032.stdout: "num_object_clones": 1, 2024-09-13T12:17:07.610 INFO:tasks.workunit.client.0.smithi032.stdout: "num_object_copies": 303, 2024-09-13T12:17:07.610 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_missing_on_primary": 0, 2024-09-13T12:17:07.610 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_missing": 0, 2024-09-13T12:17:07.610 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_degraded": 101, 2024-09-13T12:17:07.610 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_misplaced": 0, 2024-09-13T12:17:07.610 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_unfound": 0, 2024-09-13T12:17:07.610 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_dirty": 101, 2024-09-13T12:17:07.610 INFO:tasks.workunit.client.0.smithi032.stdout: "num_whiteouts": 0, 2024-09-13T12:17:07.611 INFO:tasks.workunit.client.0.smithi032.stdout: "num_read": 0, 2024-09-13T12:17:07.611 INFO:tasks.workunit.client.0.smithi032.stdout: "num_read_kb": 0, 2024-09-13T12:17:07.611 INFO:tasks.workunit.client.0.smithi032.stdout: "num_write": 101, 2024-09-13T12:17:07.611 INFO:tasks.workunit.client.0.smithi032.stdout: "num_write_kb": 101, 2024-09-13T12:17:07.611 INFO:tasks.workunit.client.0.smithi032.stdout: "num_scrub_errors": 0, 2024-09-13T12:17:07.611 INFO:tasks.workunit.client.0.smithi032.stdout: "num_shallow_scrub_errors": 0, 2024-09-13T12:17:07.611 INFO:tasks.workunit.client.0.smithi032.stdout: "num_deep_scrub_errors": 0, 2024-09-13T12:17:07.611 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_recovered": 0, 2024-09-13T12:17:07.611 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes_recovered": 0, 2024-09-13T12:17:07.611 INFO:tasks.workunit.client.0.smithi032.stdout: "num_keys_recovered": 0, 2024-09-13T12:17:07.611 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_omap": 0, 2024-09-13T12:17:07.611 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_hit_set_archive": 0, 2024-09-13T12:17:07.612 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes_hit_set_archive": 0, 2024-09-13T12:17:07.612 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush": 0, 2024-09-13T12:17:07.612 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_kb": 0, 2024-09-13T12:17:07.612 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict": 0, 2024-09-13T12:17:07.612 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_kb": 0, 2024-09-13T12:17:07.612 INFO:tasks.workunit.client.0.smithi032.stdout: "num_promote": 0, 2024-09-13T12:17:07.612 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_mode_high": 0, 2024-09-13T12:17:07.612 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_mode_low": 0, 2024-09-13T12:17:07.612 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_mode_some": 0, 2024-09-13T12:17:07.612 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_mode_full": 0, 2024-09-13T12:17:07.612 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_pinned": 0, 2024-09-13T12:17:07.612 INFO:tasks.workunit.client.0.smithi032.stdout: "num_legacy_snapsets": 0, 2024-09-13T12:17:07.613 INFO:tasks.workunit.client.0.smithi032.stdout: "num_large_omap_objects": 0, 2024-09-13T12:17:07.613 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_manifest": 0, 2024-09-13T12:17:07.613 INFO:tasks.workunit.client.0.smithi032.stdout: "num_omap_bytes": 0, 2024-09-13T12:17:07.613 INFO:tasks.workunit.client.0.smithi032.stdout: "num_omap_keys": 0, 2024-09-13T12:17:07.613 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_repaired": 0 2024-09-13T12:17:07.613 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.613 INFO:tasks.workunit.client.0.smithi032.stdout: "store_stats": { 2024-09-13T12:17:07.613 INFO:tasks.workunit.client.0.smithi032.stdout: "total": 0, 2024-09-13T12:17:07.613 INFO:tasks.workunit.client.0.smithi032.stdout: "available": 0, 2024-09-13T12:17:07.613 INFO:tasks.workunit.client.0.smithi032.stdout: "internally_reserved": 0, 2024-09-13T12:17:07.613 INFO:tasks.workunit.client.0.smithi032.stdout: "allocated": 0, 2024-09-13T12:17:07.613 INFO:tasks.workunit.client.0.smithi032.stdout: "data_stored": 0, 2024-09-13T12:17:07.613 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed": 0, 2024-09-13T12:17:07.614 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_allocated": 0, 2024-09-13T12:17:07.614 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_original": 0, 2024-09-13T12:17:07.614 INFO:tasks.workunit.client.0.smithi032.stdout: "omap_allocated": 0, 2024-09-13T12:17:07.614 INFO:tasks.workunit.client.0.smithi032.stdout: "internal_metadata": 0 2024-09-13T12:17:07.614 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.614 INFO:tasks.workunit.client.0.smithi032.stdout: "log_size": 18, 2024-09-13T12:17:07.614 INFO:tasks.workunit.client.0.smithi032.stdout: "ondisk_log_size": 18, 2024-09-13T12:17:07.614 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 3, 2024-09-13T12:17:07.614 INFO:tasks.workunit.client.0.smithi032.stdout: "acting": 3, 2024-09-13T12:17:07.614 INFO:tasks.workunit.client.0.smithi032.stdout: "num_store_stats": 0 2024-09-13T12:17:07.614 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.614 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_sum": { 2024-09-13T12:17:07.614 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 0, 2024-09-13T12:17:07.615 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 0, 2024-09-13T12:17:07.615 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pgs": 3, 2024-09-13T12:17:07.615 INFO:tasks.workunit.client.0.smithi032.stdout: "num_osds": 3, 2024-09-13T12:17:07.615 INFO:tasks.workunit.client.0.smithi032.stdout: "num_per_pool_osds": 3, 2024-09-13T12:17:07.615 INFO:tasks.workunit.client.0.smithi032.stdout: "num_per_pool_omap_osds": 3, 2024-09-13T12:17:07.615 INFO:tasks.workunit.client.0.smithi032.stdout: "kb": 314572800, 2024-09-13T12:17:07.615 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used": 85104, 2024-09-13T12:17:07.615 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_data": 2276, 2024-09-13T12:17:07.615 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_omap": 15, 2024-09-13T12:17:07.615 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_meta": 82736, 2024-09-13T12:17:07.615 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_avail": 314487696, 2024-09-13T12:17:07.615 INFO:tasks.workunit.client.0.smithi032.stdout: "statfs": { 2024-09-13T12:17:07.615 INFO:tasks.workunit.client.0.smithi032.stdout: "total": 322122547200, 2024-09-13T12:17:07.616 INFO:tasks.workunit.client.0.smithi032.stdout: "available": 322035400704, 2024-09-13T12:17:07.616 INFO:tasks.workunit.client.0.smithi032.stdout: "internally_reserved": 0, 2024-09-13T12:17:07.616 INFO:tasks.workunit.client.0.smithi032.stdout: "allocated": 2330624, 2024-09-13T12:17:07.616 INFO:tasks.workunit.client.0.smithi032.stdout: "data_stored": 1599159, 2024-09-13T12:17:07.616 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed": 0, 2024-09-13T12:17:07.616 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_allocated": 0, 2024-09-13T12:17:07.616 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_original": 0, 2024-09-13T12:17:07.616 INFO:tasks.workunit.client.0.smithi032.stdout: "omap_allocated": 16266, 2024-09-13T12:17:07.616 INFO:tasks.workunit.client.0.smithi032.stdout: "internal_metadata": 84721782 2024-09-13T12:17:07.616 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.616 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_peers": [], 2024-09-13T12:17:07.616 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_trim_queue_len": 0, 2024-09-13T12:17:07.616 INFO:tasks.workunit.client.0.smithi032.stdout: "num_snap_trimming": 0, 2024-09-13T12:17:07.616 INFO:tasks.workunit.client.0.smithi032.stdout: "num_shards_repaired": 0, 2024-09-13T12:17:07.617 INFO:tasks.workunit.client.0.smithi032.stdout: "op_queue_age_hist": { 2024-09-13T12:17:07.617 INFO:tasks.workunit.client.0.smithi032.stdout: "histogram": [], 2024-09-13T12:17:07.617 INFO:tasks.workunit.client.0.smithi032.stdout: "upper_bound": 1 2024-09-13T12:17:07.617 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.617 INFO:tasks.workunit.client.0.smithi032.stdout: "perf_stat": { 2024-09-13T12:17:07.617 INFO:tasks.workunit.client.0.smithi032.stdout: "commit_latency_ms": 0, 2024-09-13T12:17:07.617 INFO:tasks.workunit.client.0.smithi032.stdout: "apply_latency_ms": 0, 2024-09-13T12:17:07.617 INFO:tasks.workunit.client.0.smithi032.stdout: "commit_latency_ns": 0, 2024-09-13T12:17:07.617 INFO:tasks.workunit.client.0.smithi032.stdout: "apply_latency_ns": 0 2024-09-13T12:17:07.617 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.617 INFO:tasks.workunit.client.0.smithi032.stdout: "alerts": [], 2024-09-13T12:17:07.617 INFO:tasks.workunit.client.0.smithi032.stdout: "network_ping_times": [] 2024-09-13T12:17:07.617 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.617 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_sum_by_class": { 2024-09-13T12:17:07.618 INFO:tasks.workunit.client.0.smithi032.stdout: "hdd": { 2024-09-13T12:17:07.618 INFO:tasks.workunit.client.0.smithi032.stdout: "up_from": 0, 2024-09-13T12:17:07.618 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 0, 2024-09-13T12:17:07.618 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pgs": 3, 2024-09-13T12:17:07.618 INFO:tasks.workunit.client.0.smithi032.stdout: "num_osds": 3, 2024-09-13T12:17:07.618 INFO:tasks.workunit.client.0.smithi032.stdout: "num_per_pool_osds": 3, 2024-09-13T12:17:07.618 INFO:tasks.workunit.client.0.smithi032.stdout: "num_per_pool_omap_osds": 3, 2024-09-13T12:17:07.618 INFO:tasks.workunit.client.0.smithi032.stdout: "kb": 314572800, 2024-09-13T12:17:07.618 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used": 85104, 2024-09-13T12:17:07.618 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_data": 2276, 2024-09-13T12:17:07.618 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_omap": 15, 2024-09-13T12:17:07.618 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_used_meta": 82736, 2024-09-13T12:17:07.618 INFO:tasks.workunit.client.0.smithi032.stdout: "kb_avail": 314487696, 2024-09-13T12:17:07.618 INFO:tasks.workunit.client.0.smithi032.stdout: "statfs": { 2024-09-13T12:17:07.619 INFO:tasks.workunit.client.0.smithi032.stdout: "total": 322122547200, 2024-09-13T12:17:07.619 INFO:tasks.workunit.client.0.smithi032.stdout: "available": 322035400704, 2024-09-13T12:17:07.619 INFO:tasks.workunit.client.0.smithi032.stdout: "internally_reserved": 0, 2024-09-13T12:17:07.619 INFO:tasks.workunit.client.0.smithi032.stdout: "allocated": 2330624, 2024-09-13T12:17:07.619 INFO:tasks.workunit.client.0.smithi032.stdout: "data_stored": 1599159, 2024-09-13T12:17:07.619 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed": 0, 2024-09-13T12:17:07.619 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_allocated": 0, 2024-09-13T12:17:07.619 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_original": 0, 2024-09-13T12:17:07.619 INFO:tasks.workunit.client.0.smithi032.stdout: "omap_allocated": 16266, 2024-09-13T12:17:07.619 INFO:tasks.workunit.client.0.smithi032.stdout: "internal_metadata": 84721782 2024-09-13T12:17:07.619 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.619 INFO:tasks.workunit.client.0.smithi032.stdout: "hb_peers": [], 2024-09-13T12:17:07.619 INFO:tasks.workunit.client.0.smithi032.stdout: "snap_trim_queue_len": 0, 2024-09-13T12:17:07.620 INFO:tasks.workunit.client.0.smithi032.stdout: "num_snap_trimming": 0, 2024-09-13T12:17:07.620 INFO:tasks.workunit.client.0.smithi032.stdout: "num_shards_repaired": 0, 2024-09-13T12:17:07.620 INFO:tasks.workunit.client.0.smithi032.stdout: "op_queue_age_hist": { 2024-09-13T12:17:07.620 INFO:tasks.workunit.client.0.smithi032.stdout: "histogram": [], 2024-09-13T12:17:07.620 INFO:tasks.workunit.client.0.smithi032.stdout: "upper_bound": 1 2024-09-13T12:17:07.620 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.620 INFO:tasks.workunit.client.0.smithi032.stdout: "perf_stat": { 2024-09-13T12:17:07.620 INFO:tasks.workunit.client.0.smithi032.stdout: "commit_latency_ms": 0, 2024-09-13T12:17:07.620 INFO:tasks.workunit.client.0.smithi032.stdout: "apply_latency_ms": 0, 2024-09-13T12:17:07.620 INFO:tasks.workunit.client.0.smithi032.stdout: "commit_latency_ns": 0, 2024-09-13T12:17:07.620 INFO:tasks.workunit.client.0.smithi032.stdout: "apply_latency_ns": 0 2024-09-13T12:17:07.620 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.620 INFO:tasks.workunit.client.0.smithi032.stdout: "alerts": [], 2024-09-13T12:17:07.620 INFO:tasks.workunit.client.0.smithi032.stdout: "network_ping_times": [] 2024-09-13T12:17:07.621 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.621 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.621 INFO:tasks.workunit.client.0.smithi032.stdout: "pool_stats": [ 2024-09-13T12:17:07.621 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.621 INFO:tasks.workunit.client.0.smithi032.stdout: "poolid": 1, 2024-09-13T12:17:07.621 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg": 1, 2024-09-13T12:17:07.621 INFO:tasks.workunit.client.0.smithi032.stdout: "stat_sum": { 2024-09-13T12:17:07.621 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes": 48454, 2024-09-13T12:17:07.621 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects": 101, 2024-09-13T12:17:07.621 INFO:tasks.workunit.client.0.smithi032.stdout: "num_object_clones": 1, 2024-09-13T12:17:07.621 INFO:tasks.workunit.client.0.smithi032.stdout: "num_object_copies": 303, 2024-09-13T12:17:07.621 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_missing_on_primary": 0, 2024-09-13T12:17:07.621 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_missing": 0, 2024-09-13T12:17:07.621 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_degraded": 101, 2024-09-13T12:17:07.622 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_misplaced": 0, 2024-09-13T12:17:07.622 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_unfound": 0, 2024-09-13T12:17:07.622 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_dirty": 101, 2024-09-13T12:17:07.622 INFO:tasks.workunit.client.0.smithi032.stdout: "num_whiteouts": 0, 2024-09-13T12:17:07.622 INFO:tasks.workunit.client.0.smithi032.stdout: "num_read": 0, 2024-09-13T12:17:07.622 INFO:tasks.workunit.client.0.smithi032.stdout: "num_read_kb": 0, 2024-09-13T12:17:07.622 INFO:tasks.workunit.client.0.smithi032.stdout: "num_write": 101, 2024-09-13T12:17:07.622 INFO:tasks.workunit.client.0.smithi032.stdout: "num_write_kb": 101, 2024-09-13T12:17:07.622 INFO:tasks.workunit.client.0.smithi032.stdout: "num_scrub_errors": 0, 2024-09-13T12:17:07.622 INFO:tasks.workunit.client.0.smithi032.stdout: "num_shallow_scrub_errors": 0, 2024-09-13T12:17:07.622 INFO:tasks.workunit.client.0.smithi032.stdout: "num_deep_scrub_errors": 0, 2024-09-13T12:17:07.622 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_recovered": 0, 2024-09-13T12:17:07.622 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes_recovered": 0, 2024-09-13T12:17:07.622 INFO:tasks.workunit.client.0.smithi032.stdout: "num_keys_recovered": 0, 2024-09-13T12:17:07.623 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_omap": 0, 2024-09-13T12:17:07.623 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_hit_set_archive": 0, 2024-09-13T12:17:07.623 INFO:tasks.workunit.client.0.smithi032.stdout: "num_bytes_hit_set_archive": 0, 2024-09-13T12:17:07.623 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush": 0, 2024-09-13T12:17:07.623 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_kb": 0, 2024-09-13T12:17:07.623 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict": 0, 2024-09-13T12:17:07.623 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_kb": 0, 2024-09-13T12:17:07.623 INFO:tasks.workunit.client.0.smithi032.stdout: "num_promote": 0, 2024-09-13T12:17:07.623 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_mode_high": 0, 2024-09-13T12:17:07.623 INFO:tasks.workunit.client.0.smithi032.stdout: "num_flush_mode_low": 0, 2024-09-13T12:17:07.623 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_mode_some": 0, 2024-09-13T12:17:07.623 INFO:tasks.workunit.client.0.smithi032.stdout: "num_evict_mode_full": 0, 2024-09-13T12:17:07.623 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_pinned": 0, 2024-09-13T12:17:07.624 INFO:tasks.workunit.client.0.smithi032.stdout: "num_legacy_snapsets": 0, 2024-09-13T12:17:07.624 INFO:tasks.workunit.client.0.smithi032.stdout: "num_large_omap_objects": 0, 2024-09-13T12:17:07.624 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_manifest": 0, 2024-09-13T12:17:07.624 INFO:tasks.workunit.client.0.smithi032.stdout: "num_omap_bytes": 0, 2024-09-13T12:17:07.624 INFO:tasks.workunit.client.0.smithi032.stdout: "num_omap_keys": 0, 2024-09-13T12:17:07.624 INFO:tasks.workunit.client.0.smithi032.stdout: "num_objects_repaired": 0 2024-09-13T12:17:07.624 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.624 INFO:tasks.workunit.client.0.smithi032.stdout: "store_stats": { 2024-09-13T12:17:07.624 INFO:tasks.workunit.client.0.smithi032.stdout: "total": 0, 2024-09-13T12:17:07.624 INFO:tasks.workunit.client.0.smithi032.stdout: "available": 0, 2024-09-13T12:17:07.624 INFO:tasks.workunit.client.0.smithi032.stdout: "internally_reserved": 0, 2024-09-13T12:17:07.624 INFO:tasks.workunit.client.0.smithi032.stdout: "allocated": 1241088, 2024-09-13T12:17:07.624 INFO:tasks.workunit.client.0.smithi032.stdout: "data_stored": 1241088, 2024-09-13T12:17:07.624 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed": 0, 2024-09-13T12:17:07.625 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_allocated": 0, 2024-09-13T12:17:07.625 INFO:tasks.workunit.client.0.smithi032.stdout: "data_compressed_original": 0, 2024-09-13T12:17:07.625 INFO:tasks.workunit.client.0.smithi032.stdout: "omap_allocated": 0, 2024-09-13T12:17:07.625 INFO:tasks.workunit.client.0.smithi032.stdout: "internal_metadata": 0 2024-09-13T12:17:07.625 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.625 INFO:tasks.workunit.client.0.smithi032.stdout: "log_size": 18, 2024-09-13T12:17:07.625 INFO:tasks.workunit.client.0.smithi032.stdout: "ondisk_log_size": 18, 2024-09-13T12:17:07.625 INFO:tasks.workunit.client.0.smithi032.stdout: "up": 3, 2024-09-13T12:17:07.625 INFO:tasks.workunit.client.0.smithi032.stdout: "acting": 3, 2024-09-13T12:17:07.625 INFO:tasks.workunit.client.0.smithi032.stdout: "num_store_stats": 3 2024-09-13T12:17:07.625 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.625 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:17:07.625 INFO:tasks.workunit.client.0.smithi032.stdout: "osd_stats": [ 2024-09-13T12:17:07.625 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.626 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 0, 2024-09-13T12:17:07.626 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 193273528331 2024-09-13T12:17:07.626 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.626 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.626 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 1, 2024-09-13T12:17:07.626 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 42949672990 2024-09-13T12:17:07.626 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.626 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.626 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2, 2024-09-13T12:17:07.626 INFO:tasks.workunit.client.0.smithi032.stdout: "seq": 210453397514 2024-09-13T12:17:07.626 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.626 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:17:07.626 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg_by_state": [ 2024-09-13T12:17:07.626 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.626 INFO:tasks.workunit.client.0.smithi032.stdout: "state": "active+undersized+degraded", 2024-09-13T12:17:07.627 INFO:tasks.workunit.client.0.smithi032.stdout: "num": 1 2024-09-13T12:17:07.627 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.627 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:17:07.627 INFO:tasks.workunit.client.0.smithi032.stdout: "num_pg_by_osd": [ 2024-09-13T12:17:07.627 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.627 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2147483647, 2024-09-13T12:17:07.627 INFO:tasks.workunit.client.0.smithi032.stdout: "num_primary_pg": 0, 2024-09-13T12:17:07.627 INFO:tasks.workunit.client.0.smithi032.stdout: "num_acting_pg": 1, 2024-09-13T12:17:07.627 INFO:tasks.workunit.client.0.smithi032.stdout: "num_up_not_acting_pg": 0 2024-09-13T12:17:07.627 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.627 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.627 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 2, 2024-09-13T12:17:07.627 INFO:tasks.workunit.client.0.smithi032.stdout: "num_primary_pg": 0, 2024-09-13T12:17:07.628 INFO:tasks.workunit.client.0.smithi032.stdout: "num_acting_pg": 1, 2024-09-13T12:17:07.628 INFO:tasks.workunit.client.0.smithi032.stdout: "num_up_not_acting_pg": 0 2024-09-13T12:17:07.628 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.628 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.628 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 0, 2024-09-13T12:17:07.628 INFO:tasks.workunit.client.0.smithi032.stdout: "num_primary_pg": 1, 2024-09-13T12:17:07.628 INFO:tasks.workunit.client.0.smithi032.stdout: "num_acting_pg": 1, 2024-09-13T12:17:07.628 INFO:tasks.workunit.client.0.smithi032.stdout: "num_up_not_acting_pg": 0 2024-09-13T12:17:07.628 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.628 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.629 INFO:tasks.workunit.client.0.smithi032.stdout: "osd": 1, 2024-09-13T12:17:07.629 INFO:tasks.workunit.client.0.smithi032.stdout: "num_primary_pg": 0, 2024-09-13T12:17:07.629 INFO:tasks.workunit.client.0.smithi032.stdout: "num_acting_pg": 0, 2024-09-13T12:17:07.629 INFO:tasks.workunit.client.0.smithi032.stdout: "num_up_not_acting_pg": 0 2024-09-13T12:17:07.629 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.629 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:17:07.629 INFO:tasks.workunit.client.0.smithi032.stdout: "purged_snaps": [ 2024-09-13T12:17:07.629 INFO:tasks.workunit.client.0.smithi032.stdout: { 2024-09-13T12:17:07.629 INFO:tasks.workunit.client.0.smithi032.stdout: "pool": 1, 2024-09-13T12:17:07.629 INFO:tasks.workunit.client.0.smithi032.stdout: "purged_snaps": {} 2024-09-13T12:17:07.629 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.629 INFO:tasks.workunit.client.0.smithi032.stdout: ], 2024-09-13T12:17:07.629 INFO:tasks.workunit.client.0.smithi032.stdout: "servicemap": { 2024-09-13T12:17:07.629 INFO:tasks.workunit.client.0.smithi032.stdout: "epoch": 5, 2024-09-13T12:17:07.630 INFO:tasks.workunit.client.0.smithi032.stdout: "modified": "2024-09-13T12:15:20.230759+0000", 2024-09-13T12:17:07.630 INFO:tasks.workunit.client.0.smithi032.stdout: "services": {} 2024-09-13T12:17:07.630 INFO:tasks.workunit.client.0.smithi032.stdout: }, 2024-09-13T12:17:07.630 INFO:tasks.workunit.client.0.smithi032.stdout: "mgrstat_first_committed": 1, 2024-09-13T12:17:07.630 INFO:tasks.workunit.client.0.smithi032.stdout: "mgrstat_last_committed": 127, 2024-09-13T12:17:07.630 INFO:tasks.workunit.client.0.smithi032.stdout: "logm_first_committed": 1, 2024-09-13T12:17:07.630 INFO:tasks.workunit.client.0.smithi032.stdout: "logm_last_committed": 490, 2024-09-13T12:17:07.630 INFO:tasks.workunit.client.0.smithi032.stdout: "paxos": { 2024-09-13T12:17:07.630 INFO:tasks.workunit.client.0.smithi032.stdout: "first_committed": 1, 2024-09-13T12:17:07.630 INFO:tasks.workunit.client.0.smithi032.stdout: "last_committed": 676, 2024-09-13T12:17:07.630 INFO:tasks.workunit.client.0.smithi032.stdout: "last_pn": 0, 2024-09-13T12:17:07.630 INFO:tasks.workunit.client.0.smithi032.stdout: "accepted_pn": 0 2024-09-13T12:17:07.630 INFO:tasks.workunit.client.0.smithi032.stdout: } 2024-09-13T12:17:07.630 INFO:tasks.workunit.client.0.smithi032.stdout:} 2024-09-13T12:17:07.631 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-13T12:17:07.631 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2024-09-13T12:17:07.632 INFO:tasks.workunit.client.0.smithi032.stdout:delay recovery 2024-09-13T12:17:07.632 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2024-09-13T12:17:07.969 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:17:07.969 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-13T12:16:28.240918+0000 42'102 50:141 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-13T12:14:22.777057+0000 0'0 2024-09-13T12:14:22.777057+0000 0 0 periodic scrub scheduled @ 2024-09-14T16:45:01.217472+0000 0 0 2024-09-13T12:17:07.969 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:17:07.969 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:17:07.969 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:17:07.980 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-09-13T12:17:07.981 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2024-09-13T12:17:07.981 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-13T12:17:07.981 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-13T12:17:07.981 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:17:07.981 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:17:07.981 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:17:07.981 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-osd.0.asok 2024-09-13T12:17:07.981 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-09-13T12:17:07.982 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.33242/ceph-osd.0.asok set_recovery_delay 100000 2024-09-13T12:17:08.068 INFO:tasks.workunit.client.0.smithi032.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-13T12:17:08.072 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-09-13T12:17:08.073 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2024-09-13T12:17:08.073 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-13T12:17:08.073 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-13T12:17:08.073 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:17:08.074 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:17:08.074 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:17:08.074 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-osd.2.asok 2024-09-13T12:17:08.074 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-09-13T12:17:08.074 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.33242/ceph-osd.2.asok set_recovery_delay 100000 2024-09-13T12:17:08.158 INFO:tasks.workunit.client.0.smithi032.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-13T12:17:08.165 INFO:tasks.workunit.client.0.smithi032.stdout:revive divergent 1 2024-09-13T12:17:08.165 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2024-09-13T12:17:08.165 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2024-09-13T12:17:08.578 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:17:08.578 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-13T12:16:28.240918+0000 42'102 50:141 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-13T12:14:22.777057+0000 0'0 2024-09-13T12:14:22.777057+0000 0 0 periodic scrub scheduled @ 2024-09-14T16:45:01.217472+0000 0 0 2024-09-13T12:17:08.579 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:17:08.579 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:17:08.579 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:17:08.590 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2024-09-13T12:17:09.219 INFO:tasks.workunit.client.0.smithi032.stderr:noup is set 2024-09-13T12:17:09.233 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-09-13T12:17:09.233 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-13T12:17:09.233 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:17:09.233 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T12:17:09.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:17:09.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-13T12:17:09.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T12:17:09.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:17:09.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:17:09.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:17:09.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-13T12:17:09.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-13T12:17:09.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:17:09.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:17:09.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T12:17:09.234 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:17:09.234 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:17:09.234 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:17:09.235 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:17:09.235 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:17:09.235 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:17:09.235 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:17:09.235 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:17:09.235 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:17:09.235 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T12:17:09.235 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T12:17:09.235 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:17:09.235 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:17:09.235 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:17:09.235 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:17:09.235 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:17:09.235 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:17:09.236 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-13T12:17:09.236 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.1 2024-09-13T12:17:09.237 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T12:17:09.237 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:17:09.237 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-13T12:17:09.237 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T12:17:09.237 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:17:09.237 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:17:09.237 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:17:09.269 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:17:09.268+0000 7f2f8cbdd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:17:09.287 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:17:09.286+0000 7f2f8cbdd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:17:09.288 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:17:09.287+0000 7f2f8cbdd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:17:09.710 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: return 0 2024-09-13T12:17:09.710 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2024-09-13T12:17:10.131 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:17:10.130+0000 7f2f8cbdd640 -1 Falling back to public interface 2024-09-13T12:17:12.565 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:17:12.563+0000 7f2f8cbdd640 -1 osd.1 30 log_to_monitors true 2024-09-13T12:17:14.711 INFO:tasks.workunit.client.0.smithi032.stdout:delay recovery divergent 2024-09-13T12:17:14.711 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2024-09-13T12:17:14.712 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2024-09-13T12:17:15.111 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:17:15.112 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-13T12:16:28.240918+0000 42'102 51:143 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-13T12:14:22.777057+0000 0'0 2024-09-13T12:14:22.777057+0000 0 0 periodic scrub scheduled @ 2024-09-14T16:45:01.217472+0000 0 0 2024-09-13T12:17:15.112 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:17:15.112 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:17:15.112 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:17:15.123 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2024-09-13T12:17:15.124 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-13T12:17:15.124 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-13T12:17:15.124 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:17:15.124 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:17:15.124 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:17:15.124 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33242/ceph-osd.1.asok 2024-09-13T12:17:15.124 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2024-09-13T12:17:15.124 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.33242/ceph-osd.1.asok set_recovery_delay 100000 2024-09-13T12:17:15.209 INFO:tasks.workunit.client.0.smithi032.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-13T12:17:15.216 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2024-09-13T12:17:16.249 INFO:tasks.workunit.client.0.smithi032.stderr:noup is unset 2024-09-13T12:17:16.262 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2024-09-13T12:17:16.262 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:17:16.262 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T12:17:16.262 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:17:16.262 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:17:16.262 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:17:16.263 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:17:16.263 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:17:16.263 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:17:16.263 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:17:16.732 INFO:tasks.workunit.client.0.smithi032.stdout:osd.0 up in weight 1 up_from 45 up_thru 49 down_at 44 last_clean_interval [37,42) [v2:127.0.0.1:6802/387212495,v1:127.0.0.1:6803/387212495] [v2:127.0.0.1:6804/387212495,v1:127.0.0.1:6805/387212495] exists,up 92bce19c-5a5e-4360-a088-1bf366fc2ecf 2024-09-13T12:17:16.733 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:17:16.733 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:17:16.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:17:16.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2024-09-13T12:17:16.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:17:16.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T12:17:16.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:17:16.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:17:16.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:17:16.734 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:17:16.734 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:17:16.735 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:17:16.735 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:17:17.259 INFO:tasks.workunit.client.0.smithi032.stdout:osd.1 up in weight 1 up_from 55 up_thru 55 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/4027233092,v1:127.0.0.1:6819/4027233092] [v2:127.0.0.1:6820/4027233092,v1:127.0.0.1:6821/4027233092] exists,up b6fb1428-41f7-46eb-8099-15ef8f53355b 2024-09-13T12:17:17.261 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:17:17.261 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:17:17.261 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:17:17.261 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2024-09-13T12:17:17.261 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:17:17.261 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T12:17:17.261 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:17:17.261 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:17:17.261 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:17:17.261 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:17:17.262 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:17:17.262 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:17:17.262 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:17:17.681 INFO:tasks.workunit.client.0.smithi032.stdout:osd.2 up in weight 1 up_from 49 up_thru 0 down_at 48 last_clean_interval [39,42) [v2:127.0.0.1:6810/268464394,v1:127.0.0.1:6811/268464394] [v2:127.0.0.1:6812/268464394,v1:127.0.0.1:6813/268464394] exists,up 575c91a2-dd41-4141-bbec-f3711b8a1a2e 2024-09-13T12:17:17.681 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:17:17.682 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:17:17.682 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:17:17.682 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2024-09-13T12:17:18.071 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:17:18.071 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-13T12:16:28.240918+0000 42'102 53:147 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-13T12:14:22.777057+0000 0'0 2024-09-13T12:14:22.777057+0000 0 0 periodic scrub scheduled @ 2024-09-14T16:45:01.217472+0000 0 0 2024-09-13T12:17:18.071 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:17:18.071 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:17:18.071 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:17:18.082 INFO:tasks.workunit.client.0.smithi032.stdout:wait for peering 2024-09-13T12:17:18.082 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2024-09-13T12:17:18.082 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2024-09-13T12:17:18.475 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:17:18.475 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 101 4 4 0 0 48454 0 0 12 90 12 active+recovering+degraded 2024-09-13T12:17:16.796095+0000 42'102 57:136 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-13T12:14:22.777057+0000 0'0 2024-09-13T12:14:22.777057+0000 0 0 no scrub is scheduled 0 0 2024-09-13T12:17:18.475 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:17:18.475 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:17:18.476 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:17:18.486 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2024-09-13T12:17:18.602 INFO:tasks.workunit.client.0.smithi032.stdout:killing divergent 1 2024-09-13T12:17:18.603 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2024-09-13T12:17:18.603 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2024-09-13T12:17:18.991 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:17:18.991 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 101 4 4 0 0 48454 0 0 12 90 12 active+recovering+degraded 2024-09-13T12:17:16.796095+0000 42'102 57:136 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-13T12:14:22.777057+0000 0'0 2024-09-13T12:14:22.777057+0000 0 0 no scrub is scheduled 0 0 2024-09-13T12:17:18.991 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:17:18.991 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:17:18.992 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:17:19.002 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2024-09-13T12:17:19.003 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:17:19.003 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:17:19.003 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:17:19.003 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:17:19.003 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:17:19.112 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:17:19.113 INFO:tasks.workunit.client.0.smithi032.stdout:reviving divergent 1 2024-09-13T12:17:19.113 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2024-09-13T12:17:19.113 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2024-09-13T12:17:19.508 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:17:19.508 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 101 4 4 0 0 48454 0 0 12 90 12 active+recovering+degraded 2024-09-13T12:17:16.796095+0000 42'102 57:136 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-13T12:14:22.777057+0000 0'0 2024-09-13T12:14:22.777057+0000 0 0 no scrub is scheduled 0 0 2024-09-13T12:17:19.508 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:17:19.508 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:17:19.509 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:17:19.522 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-09-13T12:17:19.522 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-13T12:17:19.523 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:17:19.523 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T12:17:19.523 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:17:19.523 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-13T12:17:19.523 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-13T12:17:19.523 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:17:19.523 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:17:19.523 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:17:19.523 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-13T12:17:19.523 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-13T12:17:19.523 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:17:19.523 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:17:19.523 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-13T12:17:19.524 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:17:19.524 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:17:19.524 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:17:19.524 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:17:19.524 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:17:19.524 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:17:19.524 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:17:19.524 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' 2024-09-13T12:17:19.524 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:17:19.524 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-13T12:17:19.525 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-13T12:17:19.525 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:17:19.525 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:17:19.525 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:17:19.525 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:17:19.525 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:17:19.525 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:17:19.525 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-13T12:17:19.525 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.1 2024-09-13T12:17:19.526 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T12:17:19.527 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e12dfbc3-9bf5-45db-a6f0-f25db93ee351 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33242/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:17:19.527 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-13T12:17:19.527 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T12:17:19.527 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:17:19.527 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:17:19.527 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:17:19.558 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:17:19.557+0000 7fd27e71b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:17:19.560 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:17:19.559+0000 7fd27e71b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:17:19.561 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:17:19.560+0000 7fd27e71b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:17:19.965 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T12:17:19.966 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:17:19.966 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T12:17:19.966 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:17:19.966 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:17:19.966 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:17:19.966 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:17:19.967 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:17:19.967 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:17:19.967 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:17:20.440 INFO:tasks.workunit.client.0.smithi032.stdout:osd.1 up in weight 1 up_from 55 up_thru 55 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/4027233092,v1:127.0.0.1:6819/4027233092] [v2:127.0.0.1:6820/4027233092,v1:127.0.0.1:6821/4027233092] exists,up b6fb1428-41f7-46eb-8099-15ef8f53355b 2024-09-13T12:17:20.441 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:17:20.441 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:17:20.441 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:17:20.441 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2024-09-13T12:17:20.661 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:17:20.660+0000 7fd27e71b640 -1 Falling back to public interface 2024-09-13T12:17:23.714 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:17:23.713+0000 7fd27e71b640 -1 osd.1 57 log_to_monitors true 2024-09-13T12:17:40.442 INFO:tasks.workunit.client.0.smithi032.stdout:allowing recovery 2024-09-13T12:17:40.443 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2024-09-13T12:17:40.443 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2024-09-13T12:17:40.849 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:17:40.849 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 102 0 0 0 0 48937 0 0 13 90 13 active+clean 2024-09-13T12:17:26.039071+0000 57'103 62:161 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-13T12:14:22.777057+0000 0'0 2024-09-13T12:14:22.777057+0000 0 0 periodic scrub scheduled @ 2024-09-14T18:05:19.335436+0000 0 0 2024-09-13T12:17:40.849 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:17:40.849 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:17:40.849 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:17:40.860 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-09-13T12:17:40.861 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-13T12:17:40.977 INFO:tasks.workunit.client.0.smithi032.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-13T12:17:40.986 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-09-13T12:17:40.986 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-13T12:17:41.102 INFO:tasks.workunit.client.0.smithi032.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-13T12:17:41.111 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-09-13T12:17:41.111 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-13T12:17:41.244 INFO:tasks.workunit.client.0.smithi032.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-13T12:17:41.252 INFO:tasks.workunit.client.0.smithi032.stdout:reading divergent objects 2024-09-13T12:17:41.253 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2024-09-13T12:17:41.253 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2024-09-13T12:17:41.641 INFO:tasks.workunit.client.0.smithi032.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T12:17:41.641 INFO:tasks.workunit.client.0.smithi032.stdout:1.0 102 0 0 0 0 48937 0 0 13 90 13 active+clean 2024-09-13T12:17:26.039071+0000 57'103 62:161 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-13T12:14:22.777057+0000 0'0 2024-09-13T12:14:22.777057+0000 0 0 periodic scrub scheduled @ 2024-09-14T18:05:19.335436+0000 0 0 2024-09-13T12:17:41.641 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-09-13T12:17:41.641 INFO:tasks.workunit.client.0.smithi032.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T12:17:41.641 INFO:tasks.workunit.client.0.smithi032.stderr:dumped pgs 2024-09-13T12:17:41.652 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2024-09-13T12:17:41.653 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2024-09-13T12:17:41.655 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-13T12:17:41.655 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2024-09-13T12:17:41.692 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-13T12:17:41.692 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2024-09-13T12:17:41.729 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-13T12:17:41.729 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2024-09-13T12:17:41.766 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-13T12:17:41.766 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2024-09-13T12:17:41.803 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2024-09-13T12:17:41.804 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-09-13T12:17:41.805 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-13T12:17:41.811 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:17:16.525+0000 7f2f70b4f640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-09-13T12:15:01.890874+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-13T12:17:41.811 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:17:16.525+0000 7f2f70b4f640 20 _merge_object_divergent_entries: keeping 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-09-13T12:15:01.890874+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-13T12:17:41.811 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:17:16.525+0000 7f2f70b4f640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-13T12:17:41.811 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:17:16.525+0000 7f2f70b4f640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 29'99 first_divergent_update: 30'103 last_divergent_update: 30'103 2024-09-13T12:17:41.811 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:17:16.525+0000 7f2f70b4f640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2024-09-13T12:17:41.811 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:17:16.525+0000 7f2f70b4f640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2024-09-13T12:17:41.811 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:17:16.525+0000 7f2f70b4f640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4657.0:1 2024-09-13T12:15:09.852645+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-13T12:17:41.811 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:17:16.525+0000 7f2f70b4f640 20 _merge_object_divergent_entries: keeping 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4657.0:1 2024-09-13T12:15:09.852645+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-13T12:17:41.811 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:17:16.525+0000 7f2f70b4f640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-13T12:17:41.811 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:17:16.525+0000 7f2f70b4f640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 29'99 first_divergent_update: 30'104 last_divergent_update: 30'104 2024-09-13T12:17:41.811 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:17:16.525+0000 7f2f70b4f640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2024-09-13T12:17:41.812 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:17:16.525+0000 7f2f70b4f640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2024-09-13T12:17:41.812 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:17:16.525+0000 7f2f70b4f640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 30'104 2024-09-13T12:17:41.812 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:17:16.525+0000 7f2f70b4f640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4657.0:1 2024-09-13T12:15:09.852645+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-13T12:17:41.812 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:17:16.525+0000 7f2f70b4f640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2024-09-13T12:17:41.812 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:17:16.525+0000 7f2f70b4f640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4648.0:1 2024-09-13T12:15:08.614628+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false,29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4654.0:1 2024-09-13T12:15:09.622739+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-13T12:17:41.813 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:17:16.525+0000 7f2f70b4f640 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4648.0:1 2024-09-13T12:15:08.614628+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-13T12:17:41.813 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:17:16.525+0000 7f2f70b4f640 20 _merge_object_divergent_entries: keeping 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4654.0:1 2024-09-13T12:15:09.622739+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-13T12:17:41.813 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:17:16.525+0000 7f2f70b4f640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-13T12:17:41.813 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:17:16.525+0000 7f2f70b4f640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'102 2024-09-13T12:17:41.813 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:17:16.525+0000 7f2f70b4f640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-09-13T12:17:41.813 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:17:16.525+0000 7f2f70b4f640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-09-13T12:17:41.813 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:17:16.525+0000 7f2f70b4f640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 30'104 2024-09-13T12:17:41.813 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:17:16.525+0000 7f2f70b4f640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4654.0:1 2024-09-13T12:15:09.622739+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-13T12:17:41.813 INFO:tasks.workunit.client.0.smithi032.stdout:td/divergent-priors/osd.1.log:2024-09-13T12:17:16.525+0000 7f2f70b4f640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-09-13T12:17:41.813 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-09-13T12:17:41.814 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-13T12:17:41.820 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2024-09-13T12:17:41.820 INFO:tasks.workunit.client.0.smithi032.stdout:success 2024-09-13T12:17:41.821 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2024-09-13T12:17:41.821 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test 2024-09-13T12:17:41.821 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-13T12:17:42.441 INFO:tasks.workunit.client.0.smithi032.stderr:pool 'test' does not exist 2024-09-13T12:17:42.453 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2024-09-13T12:17:42.453 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:17:42.454 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:17:42.454 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:17:42.454 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:17:42.454 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:17:43.765 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:17:43.765 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-13T12:17:43.765 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-13T12:17:43.765 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T12:17:43.765 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-13T12:17:43.765 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:17:43.765 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:17:43.766 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:17:43.766 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:17:43.766 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:17:43.772 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:17:43.773 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T12:17:43.774 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T12:17:43.774 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T12:17:43.775 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T12:17:43.775 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T12:17:43.776 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T12:17:43.777 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:17:43.777 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T12:17:43.777 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T12:17:43.778 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:17:43.778 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:17:43.780 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T12:17:43.780 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-13T12:17:43.805 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T12:17:43.805 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:17:43.805 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:17:43.805 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.33242 2024-09-13T12:17:43.807 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T12:17:43.807 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T12:17:43.807 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-13T12:17:43.807 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/divergent-priors 0 2024-09-13T12:17:43.807 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-13T12:17:43.807 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-13T12:17:43.807 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-13T12:17:43.807 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:17:43.807 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:17:43.808 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:17:43.808 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:17:43.808 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:17:43.809 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:17:43.810 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T12:17:43.811 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T12:17:43.811 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T12:17:43.812 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T12:17:43.812 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T12:17:43.812 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T12:17:43.813 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:17:43.813 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T12:17:43.814 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T12:17:43.814 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:17:43.815 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:17:43.816 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-13T12:17:43.816 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-13T12:17:43.818 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T12:17:43.818 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:17:43.818 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33242 2024-09-13T12:17:43.818 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.33242 2024-09-13T12:17:43.819 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T12:17:43.820 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T12:17:43.820 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-13T12:17:43.820 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-13T12:17:43.821 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-13T12:17:43.851 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2024-09-13T12:17:43.851 DEBUG:teuthology.orchestra.run.smithi032:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d58f73069a8157a9ace12a0247ee3e8497707ea9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2024-09-13T12:17:43.916 INFO:tasks.workunit.client.0.smithi032.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-13T12:17:43.922 INFO:tasks.workunit.client.0.smithi032.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-13T12:17:43.922 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-13T12:17:43.922 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-13T12:17:43.922 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-13T12:17:43.922 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-13T12:17:43.922 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-13T12:17:43.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-13T12:17:43.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-13T12:17:43.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-13T12:17:43.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/ec-error-rollforward 2024-09-13T12:17:43.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2024-09-13T12:17:43.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2024-09-13T12:17:43.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2024-09-13T12:17:43.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2024-09-13T12:17:43.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2024-09-13T12:17:43.923 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2024-09-13T12:17:43.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=7fa1b90f-27d7-414a-8403-1b3eb21a3b85 --auth-supported=none ' 2024-09-13T12:17:43.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2024-09-13T12:17:43.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2024-09-13T12:17:43.924 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2024-09-13T12:17:43.924 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2024-09-13T12:17:43.924 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2024-09-13T12:17:43.924 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2024-09-13T12:17:43.924 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2024-09-13T12:17:43.924 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2024-09-13T12:17:43.924 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-13T12:17:43.926 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2024-09-13T12:17:43.926 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2024-09-13T12:17:43.926 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2024-09-13T12:17:43.926 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ec-error-rollforward 2024-09-13T12:17:43.926 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ec-error-rollforward 2024-09-13T12:17:43.926 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ec-error-rollforward 2024-09-13T12:17:43.926 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T12:17:43.926 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ec-error-rollforward KILL 2024-09-13T12:17:43.926 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:17:43.926 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:17:43.926 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:17:43.926 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:17:43.926 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:17:43.927 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:17:43.927 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T12:17:43.927 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T12:17:43.927 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T12:17:43.928 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T12:17:43.928 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T12:17:43.928 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T12:17:43.929 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:17:43.929 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T12:17:43.929 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T12:17:43.929 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:17:43.930 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:17:43.933 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T12:17:43.933 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ec-error-rollforward 2024-09-13T12:17:43.933 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T12:17:43.933 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:17:43.933 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90678 2024-09-13T12:17:43.933 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.90678 2024-09-13T12:17:43.934 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T12:17:43.934 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T12:17:43.934 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ec-error-rollforward 2024-09-13T12:17:43.935 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T12:17:43.935 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:17:43.935 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90678 2024-09-13T12:17:43.935 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.90678 2024-09-13T12:17:43.937 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T12:17:43.937 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-13T12:17:43.937 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-13T12:17:43.937 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T12:17:43.937 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2024-09-13T12:17:43.938 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2024-09-13T12:17:43.938 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2024-09-13T12:17:43.938 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2024-09-13T12:17:43.938 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ec-error-rollforward 2024-09-13T12:17:43.938 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T12:17:43.938 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T12:17:43.938 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T12:17:43.938 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ec-error-rollforward/a 2024-09-13T12:17:43.938 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2024-09-13T12:17:44.398 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T12:17:44.398 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:17:44.398 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:17:44.398 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:17:44.398 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:17:44.398 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90678 2024-09-13T12:17:44.398 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.90678/$cluster-$name.asok' 2024-09-13T12:17:44.398 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.90678/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-13T12:17:44.615 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T12:17:44.617 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T12:17:44.617 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T12:17:44.617 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T12:17:44.617 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T12:17:44.617 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T12:17:44.617 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T12:17:44.617 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T12:17:44.617 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T12:17:44.618 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:17:44.618 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:17:44.618 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90678 2024-09-13T12:17:44.618 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.90678/ceph-mon.a.asok 2024-09-13T12:17:44.618 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T12:17:44.618 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.90678/ceph-mon.a.asok config get fsid 2024-09-13T12:17:44.710 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T12:17:44.710 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T12:17:44.710 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T12:17:44.710 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T12:17:44.710 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T12:17:44.710 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T12:17:44.710 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T12:17:44.710 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T12:17:44.710 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:17:44.711 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:17:44.711 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90678 2024-09-13T12:17:44.711 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.90678/ceph-mon.a.asok 2024-09-13T12:17:44.711 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T12:17:44.711 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.90678/ceph-mon.a.asok config get mon_host 2024-09-13T12:17:44.793 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2024-09-13T12:17:44.793 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ec-error-rollforward 2024-09-13T12:17:44.793 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T12:17:44.793 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T12:17:44.793 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T12:17:44.794 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ec-error-rollforward/x 2024-09-13T12:17:44.794 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T12:17:45.647 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T12:17:45.647 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:17:45.647 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:17:45.647 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:17:45.647 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:17:45.647 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90678 2024-09-13T12:17:45.647 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.90678/$cluster-$name.asok' 2024-09-13T12:17:45.647 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T12:17:45.648 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.90678/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T12:17:45.678 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2024-09-13T12:17:45.679 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ec-error-rollforward 2024-09-13T12:17:45.679 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T12:17:45.679 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T12:17:45.679 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T12:17:45.679 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ec-error-rollforward/0 2024-09-13T12:17:45.679 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7fa1b90f-27d7-414a-8403-1b3eb21a3b85 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-13T12:17:45.679 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:17:45.679 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:17:45.680 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:17:45.680 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-09-13T12:17:45.680 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-09-13T12:17:45.680 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T12:17:45.680 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T12:17:45.680 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-13T12:17:45.680 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T12:17:45.680 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:17:45.680 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:17:45.681 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:17:45.681 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:17:45.681 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90678 2024-09-13T12:17:45.681 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.90678/$cluster-$name.asok' 2024-09-13T12:17:45.681 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90678/$cluster-$name.asok' 2024-09-13T12:17:45.681 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:17:45.681 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T12:17:45.681 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T12:17:45.682 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-13T12:17:45.682 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-13T12:17:45.682 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:17:45.682 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:17:45.682 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:17:45.682 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:17:45.682 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T12:17:45.682 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T12:17:45.682 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ec-error-rollforward/0 2024-09-13T12:17:45.683 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T12:17:45.683 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2793142e-b2d9-4bef-88dc-a6fc3289aaaf 2024-09-13T12:17:45.683 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 2793142e-b2d9-4bef-88dc-a6fc3289aaaf' 2024-09-13T12:17:45.683 INFO:tasks.workunit.client.0.smithi032.stdout:add osd0 2793142e-b2d9-4bef-88dc-a6fc3289aaaf 2024-09-13T12:17:45.683 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T12:17:45.696 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBpLeRmF0psKRAAiZMxmdpH8XbonHHmg3rDvQ== 2024-09-13T12:17:45.696 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBpLeRmF0psKRAAiZMxmdpH8XbonHHmg3rDvQ=="}' 2024-09-13T12:17:45.696 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2793142e-b2d9-4bef-88dc-a6fc3289aaaf -i td/ec-error-rollforward/0/new.json 2024-09-13T12:17:45.972 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:17:45.984 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ec-error-rollforward/0/new.json 2024-09-13T12:17:45.984 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=7fa1b90f-27d7-414a-8403-1b3eb21a3b85 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90678/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBpLeRmF0psKRAAiZMxmdpH8XbonHHmg3rDvQ== --osd-uuid 2793142e-b2d9-4bef-88dc-a6fc3289aaaf 2024-09-13T12:17:46.011 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:17:46.009+0000 7fe8459b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:17:46.041 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:17:46.039+0000 7fe8459b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:17:46.042 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:17:46.040+0000 7fe8459b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:17:46.042 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:17:46.040+0000 7fe8459b2640 -1 bdev(0x557bc8087c00 td/ec-error-rollforward/0/block) open stat got: (1) Operation not permitted 2024-09-13T12:17:46.042 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:17:46.041+0000 7fe8459b2640 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2024-09-13T12:17:52.036 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2024-09-13T12:17:52.037 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T12:17:52.037 INFO:tasks.workunit.client.0.smithi032.stdout:adding osd0 key to auth repository 2024-09-13T12:17:52.038 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T12:17:52.038 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T12:17:52.598 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.0 2024-09-13T12:17:52.599 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T12:17:52.599 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=7fa1b90f-27d7-414a-8403-1b3eb21a3b85 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90678/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:17:52.599 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T12:17:52.600 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T12:17:52.600 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T12:17:52.624 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:17:52.622+0000 7f9dfdd69640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:17:52.630 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:17:52.628+0000 7f9dfdd69640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:17:52.631 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:17:52.629+0000 7f9dfdd69640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:17:53.060 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T12:17:53.060 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:17:53.060 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T12:17:53.060 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:17:53.060 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:17:53.061 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:17:53.061 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:17:53.061 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:17:53.061 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:17:53.061 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:17:53.494 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:17:53.735 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:17:53.734+0000 7f9dfdd69640 -1 Falling back to public interface 2024-09-13T12:17:54.496 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:17:54.496 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:17:54.496 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:17:54.496 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:17:54.496 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:17:54.497 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:17:54.937 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:17:55.938 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:17:55.938 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:17:55.939 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:17:55.939 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:17:55.939 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:17:55.939 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:17:56.343 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:17:56.449 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:17:56.448+0000 7f9dfdd69640 -1 osd.0 0 log_to_monitors true 2024-09-13T12:17:57.344 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:17:57.345 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:17:57.345 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:17:57.345 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:17:57.346 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:17:57.346 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:17:57.790 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:17:58.791 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:17:58.792 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:17:58.792 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:17:58.792 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:17:58.792 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:17:58.793 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:17:59.265 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:00.267 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:00.267 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:00.267 INFO:tasks.workunit.client.0.smithi032.stdout:5 2024-09-13T12:18:00.268 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T12:18:00.268 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:00.268 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:18:00.679 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:01.681 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:01.681 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:01.681 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T12:18:01.681 INFO:tasks.workunit.client.0.smithi032.stdout:6 2024-09-13T12:18:01.682 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:01.682 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:18:02.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:03.098 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:03.098 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:03.098 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T12:18:03.098 INFO:tasks.workunit.client.0.smithi032.stdout:7 2024-09-13T12:18:03.099 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:03.099 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:18:03.608 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:04.609 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:04.609 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:04.609 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T12:18:04.609 INFO:tasks.workunit.client.0.smithi032.stdout:8 2024-09-13T12:18:04.610 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:04.610 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:18:05.028 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:06.029 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:06.030 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:06.030 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T12:18:06.030 INFO:tasks.workunit.client.0.smithi032.stdout:9 2024-09-13T12:18:06.030 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:06.030 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:18:06.443 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:07.444 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:07.444 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:07.445 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T12:18:07.445 INFO:tasks.workunit.client.0.smithi032.stdout:10 2024-09-13T12:18:07.445 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:07.445 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:18:07.855 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:08.857 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:08.857 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:08.857 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T12:18:08.857 INFO:tasks.workunit.client.0.smithi032.stdout:11 2024-09-13T12:18:08.858 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:08.858 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:18:09.362 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:10.364 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:10.364 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:10.364 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T12:18:10.364 INFO:tasks.workunit.client.0.smithi032.stdout:12 2024-09-13T12:18:10.365 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:10.365 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:18:11.005 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:12.006 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:12.006 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:12.006 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T12:18:12.007 INFO:tasks.workunit.client.0.smithi032.stdout:13 2024-09-13T12:18:12.007 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:12.007 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:18:13.019 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:14.020 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:14.020 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:14.021 INFO:tasks.workunit.client.0.smithi032.stdout:14 2024-09-13T12:18:14.021 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T12:18:14.021 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:14.021 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:18:14.721 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:15.723 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:15.723 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:15.723 INFO:tasks.workunit.client.0.smithi032.stdout:15 2024-09-13T12:18:15.724 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T12:18:15.724 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:15.724 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:18:16.757 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:17.758 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:17.758 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:17.758 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T12:18:17.758 INFO:tasks.workunit.client.0.smithi032.stdout:16 2024-09-13T12:18:17.759 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:17.759 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:18:18.394 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:19.395 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:19.395 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:19.396 INFO:tasks.workunit.client.0.smithi032.stdout:17 2024-09-13T12:18:19.396 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T12:18:19.396 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:19.396 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:18:19.889 INFO:tasks.workunit.client.0.smithi032.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1038799336,v1:127.0.0.1:6803/1038799336] [v2:127.0.0.1:6804/1038799336,v1:127.0.0.1:6805/1038799336] exists,up 2793142e-b2d9-4bef-88dc-a6fc3289aaaf 2024-09-13T12:18:19.893 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:18:19.893 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:18:19.893 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:18:19.893 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2024-09-13T12:18:19.893 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ec-error-rollforward 2024-09-13T12:18:19.893 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T12:18:19.893 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T12:18:19.894 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T12:18:19.894 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ec-error-rollforward/1 2024-09-13T12:18:19.894 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7fa1b90f-27d7-414a-8403-1b3eb21a3b85 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-13T12:18:19.894 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:18:19.894 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:18:19.894 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:18:19.894 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-09-13T12:18:19.894 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-09-13T12:18:19.894 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T12:18:19.894 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T12:18:19.894 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-13T12:18:19.894 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T12:18:19.894 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:18:19.895 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:18:19.895 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:18:19.895 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:18:19.895 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90678 2024-09-13T12:18:19.895 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.90678/$cluster-$name.asok' 2024-09-13T12:18:19.895 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90678/$cluster-$name.asok' 2024-09-13T12:18:19.895 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:18:19.895 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T12:18:19.895 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T12:18:19.895 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-13T12:18:19.895 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-13T12:18:19.895 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:18:19.895 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:18:19.895 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:18:19.896 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:18:19.896 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T12:18:19.896 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T12:18:19.896 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ec-error-rollforward/1 2024-09-13T12:18:19.896 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T12:18:19.896 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ea4118e8-a0ea-4a26-8a3c-16f74116fed1 2024-09-13T12:18:19.896 INFO:tasks.workunit.client.0.smithi032.stdout:add osd1 ea4118e8-a0ea-4a26-8a3c-16f74116fed1 2024-09-13T12:18:19.897 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 ea4118e8-a0ea-4a26-8a3c-16f74116fed1' 2024-09-13T12:18:19.897 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T12:18:19.910 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCLLeRmttQwNhAA7Tk3lqQdFHIWfELac+3APA== 2024-09-13T12:18:19.910 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCLLeRmttQwNhAA7Tk3lqQdFHIWfELac+3APA=="}' 2024-09-13T12:18:19.910 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ea4118e8-a0ea-4a26-8a3c-16f74116fed1 -i td/ec-error-rollforward/1/new.json 2024-09-13T12:18:20.386 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:18:20.397 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ec-error-rollforward/1/new.json 2024-09-13T12:18:20.400 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=7fa1b90f-27d7-414a-8403-1b3eb21a3b85 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90678/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCLLeRmttQwNhAA7Tk3lqQdFHIWfELac+3APA== --osd-uuid ea4118e8-a0ea-4a26-8a3c-16f74116fed1 2024-09-13T12:18:20.425 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:18:20.424+0000 7fdfc752e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:18:20.431 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:18:20.431+0000 7fdfc752e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:18:20.432 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:18:20.432+0000 7fdfc752e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:18:20.433 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:18:20.432+0000 7fdfc752e640 -1 bdev(0x5610ab17fc00 td/ec-error-rollforward/1/block) open stat got: (1) Operation not permitted 2024-09-13T12:18:20.433 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:18:20.432+0000 7fdfc752e640 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2024-09-13T12:18:26.730 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2024-09-13T12:18:26.730 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T12:18:26.731 INFO:tasks.workunit.client.0.smithi032.stdout:adding osd1 key to auth repository 2024-09-13T12:18:26.731 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T12:18:26.731 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T12:18:27.294 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.1 2024-09-13T12:18:27.295 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T12:18:27.295 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=7fa1b90f-27d7-414a-8403-1b3eb21a3b85 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90678/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:18:27.295 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T12:18:27.295 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T12:18:27.295 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T12:18:27.320 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:18:27.319+0000 7f1bd6dcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:18:27.328 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:18:27.327+0000 7f1bd6dcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:18:27.329 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:18:27.328+0000 7f1bd6dcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:18:27.721 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T12:18:27.721 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:18:27.721 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T12:18:27.721 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:18:27.721 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:18:27.721 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:27.722 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:18:27.722 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:18:27.722 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:27.722 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:18:28.126 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:28.674 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:18:28.672+0000 7f1bd6dcc640 -1 Falling back to public interface 2024-09-13T12:18:29.128 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:29.128 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:29.128 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:18:29.128 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:18:29.128 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:29.129 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:18:29.570 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:30.572 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:30.572 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:30.572 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:18:30.573 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:18:30.573 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:30.573 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:18:30.996 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:31.617 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:18:31.615+0000 7f1bd6dcc640 -1 osd.1 0 log_to_monitors true 2024-09-13T12:18:31.998 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:31.998 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:31.998 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:18:31.998 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:18:31.999 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:31.999 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:18:32.476 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:33.475 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:33.476 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:33.476 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:18:33.476 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:18:33.476 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:33.477 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:18:33.887 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:34.888 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:34.888 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:34.889 INFO:tasks.workunit.client.0.smithi032.stdout:5 2024-09-13T12:18:34.889 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T12:18:34.889 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:34.889 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:18:35.299 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:36.300 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:36.300 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:36.300 INFO:tasks.workunit.client.0.smithi032.stdout:6 2024-09-13T12:18:36.301 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T12:18:36.301 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:36.301 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:18:36.774 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:37.775 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:37.775 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:37.775 INFO:tasks.workunit.client.0.smithi032.stdout:7 2024-09-13T12:18:37.776 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T12:18:37.776 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:37.776 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:18:38.249 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:39.250 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:39.251 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:39.251 INFO:tasks.workunit.client.0.smithi032.stdout:8 2024-09-13T12:18:39.251 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T12:18:39.251 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:39.251 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:18:39.790 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:40.792 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:40.792 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:40.792 INFO:tasks.workunit.client.0.smithi032.stdout:9 2024-09-13T12:18:40.793 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T12:18:40.793 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:40.793 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:18:41.296 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:42.297 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:42.298 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:42.298 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T12:18:42.298 INFO:tasks.workunit.client.0.smithi032.stdout:10 2024-09-13T12:18:42.298 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:42.298 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:18:42.824 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:43.825 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:43.825 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:43.826 INFO:tasks.workunit.client.0.smithi032.stdout:11 2024-09-13T12:18:43.826 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T12:18:43.826 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:43.826 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:18:44.404 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:45.406 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:45.406 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:45.406 INFO:tasks.workunit.client.0.smithi032.stdout:12 2024-09-13T12:18:45.407 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T12:18:45.407 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:45.407 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:18:45.877 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:46.878 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:46.878 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:46.879 INFO:tasks.workunit.client.0.smithi032.stdout:13 2024-09-13T12:18:46.879 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T12:18:46.879 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:46.879 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:18:47.655 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:48.656 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:48.656 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:48.657 INFO:tasks.workunit.client.0.smithi032.stdout:14 2024-09-13T12:18:48.657 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T12:18:48.657 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:48.657 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:18:49.690 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:50.691 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:50.691 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:50.691 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T12:18:50.692 INFO:tasks.workunit.client.0.smithi032.stdout:15 2024-09-13T12:18:50.692 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:50.692 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:18:51.426 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:52.427 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:52.427 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:52.427 INFO:tasks.workunit.client.0.smithi032.stdout:16 2024-09-13T12:18:52.428 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T12:18:52.428 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:52.428 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:18:53.195 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:54.197 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:54.197 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:54.197 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T12:18:54.198 INFO:tasks.workunit.client.0.smithi032.stdout:17 2024-09-13T12:18:54.198 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:54.198 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:18:54.667 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:18:55.668 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:18:55.669 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:18:55.669 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T12:18:55.669 INFO:tasks.workunit.client.0.smithi032.stdout:18 2024-09-13T12:18:55.669 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:18:55.669 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:18:56.089 INFO:tasks.workunit.client.0.smithi032.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3503804840,v1:127.0.0.1:6811/3503804840] [v2:127.0.0.1:6812/3503804840,v1:127.0.0.1:6813/3503804840] exists,up ea4118e8-a0ea-4a26-8a3c-16f74116fed1 2024-09-13T12:18:56.093 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:18:56.093 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:18:56.093 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:18:56.093 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2024-09-13T12:18:56.093 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ec-error-rollforward 2024-09-13T12:18:56.094 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T12:18:56.094 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T12:18:56.094 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T12:18:56.094 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ec-error-rollforward/2 2024-09-13T12:18:56.094 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7fa1b90f-27d7-414a-8403-1b3eb21a3b85 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-13T12:18:56.094 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:18:56.094 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:18:56.094 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:18:56.094 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-09-13T12:18:56.094 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-09-13T12:18:56.094 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T12:18:56.094 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T12:18:56.094 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-13T12:18:56.094 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T12:18:56.095 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:18:56.095 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:18:56.095 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:18:56.095 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:18:56.095 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90678 2024-09-13T12:18:56.095 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.90678/$cluster-$name.asok' 2024-09-13T12:18:56.095 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90678/$cluster-$name.asok' 2024-09-13T12:18:56.095 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:18:56.095 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T12:18:56.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T12:18:56.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-13T12:18:56.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-13T12:18:56.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:18:56.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:18:56.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:18:56.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:18:56.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T12:18:56.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T12:18:56.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ec-error-rollforward/2 2024-09-13T12:18:56.096 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T12:18:56.097 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=acb74e48-8aaf-44e5-8f27-a93216246cf7 2024-09-13T12:18:56.097 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 acb74e48-8aaf-44e5-8f27-a93216246cf7' 2024-09-13T12:18:56.097 INFO:tasks.workunit.client.0.smithi032.stdout:add osd2 acb74e48-8aaf-44e5-8f27-a93216246cf7 2024-09-13T12:18:56.097 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T12:18:56.111 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCwLeRmqASTBhAAJXaQ89gLP5VBEnIEM8iYNw== 2024-09-13T12:18:56.111 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCwLeRmqASTBhAAJXaQ89gLP5VBEnIEM8iYNw=="}' 2024-09-13T12:18:56.111 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new acb74e48-8aaf-44e5-8f27-a93216246cf7 -i td/ec-error-rollforward/2/new.json 2024-09-13T12:18:56.587 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:18:56.599 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ec-error-rollforward/2/new.json 2024-09-13T12:18:56.600 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=7fa1b90f-27d7-414a-8403-1b3eb21a3b85 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90678/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCwLeRmqASTBhAAJXaQ89gLP5VBEnIEM8iYNw== --osd-uuid acb74e48-8aaf-44e5-8f27-a93216246cf7 2024-09-13T12:18:56.628 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:18:56.627+0000 7f1f621a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:18:56.644 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:18:56.643+0000 7f1f621a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:18:56.645 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:18:56.644+0000 7f1f621a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:18:56.645 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:18:56.645+0000 7f1f621a2640 -1 bdev(0x555c5ba21c00 td/ec-error-rollforward/2/block) open stat got: (1) Operation not permitted 2024-09-13T12:18:56.645 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:18:56.645+0000 7f1f621a2640 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2024-09-13T12:19:02.619 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2024-09-13T12:19:02.619 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T12:19:02.620 INFO:tasks.workunit.client.0.smithi032.stdout:adding osd2 key to auth repository 2024-09-13T12:19:02.621 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T12:19:02.621 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T12:19:03.178 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.2 2024-09-13T12:19:03.179 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T12:19:03.179 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=7fa1b90f-27d7-414a-8403-1b3eb21a3b85 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90678/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:19:03.179 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T12:19:03.179 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T12:19:03.179 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T12:19:03.204 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:19:03.203+0000 7f6fda00b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:19:03.224 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:19:03.224+0000 7f6fda00b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:19:03.225 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:19:03.225+0000 7f6fda00b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:19:03.655 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T12:19:03.655 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:19:03.655 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T12:19:03.655 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:19:03.656 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:19:03.656 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:19:03.656 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:03.656 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:19:03.657 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:03.657 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:19:04.053 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:04.843 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:19:04.842+0000 7f6fda00b640 -1 Falling back to public interface 2024-09-13T12:19:05.054 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:05.054 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:05.054 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:19:05.055 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:19:05.055 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:05.055 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:19:05.500 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:06.501 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:06.501 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:06.501 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:19:06.502 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:19:06.502 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:06.502 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:19:06.947 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:07.623 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:19:07.622+0000 7f6fda00b640 -1 osd.2 0 log_to_monitors true 2024-09-13T12:19:07.948 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:07.948 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:07.948 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:19:07.949 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:19:07.949 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:07.949 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:19:08.446 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:09.447 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:09.447 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:09.448 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:19:09.448 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:19:09.448 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:09.448 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:19:09.873 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:10.875 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:10.875 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:10.875 INFO:tasks.workunit.client.0.smithi032.stdout:5 2024-09-13T12:19:10.876 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T12:19:10.876 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:10.876 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:19:11.425 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:12.426 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:12.426 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:12.426 INFO:tasks.workunit.client.0.smithi032.stdout:6 2024-09-13T12:19:12.427 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T12:19:12.427 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:12.427 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:19:12.836 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:13.838 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:13.838 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:13.838 INFO:tasks.workunit.client.0.smithi032.stdout:7 2024-09-13T12:19:13.839 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T12:19:13.839 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:13.839 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:19:14.267 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:15.269 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:15.269 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:15.269 INFO:tasks.workunit.client.0.smithi032.stdout:8 2024-09-13T12:19:15.270 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T12:19:15.270 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:15.270 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:19:15.741 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:16.742 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:16.742 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:16.743 INFO:tasks.workunit.client.0.smithi032.stdout:9 2024-09-13T12:19:16.743 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T12:19:16.743 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:16.743 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:19:17.181 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:18.183 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:18.183 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:18.183 INFO:tasks.workunit.client.0.smithi032.stdout:10 2024-09-13T12:19:18.184 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T12:19:18.184 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:18.184 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:19:18.600 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:19.601 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:19.601 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:19.601 INFO:tasks.workunit.client.0.smithi032.stdout:11 2024-09-13T12:19:19.602 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T12:19:19.602 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:19.602 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:19:20.075 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:21.076 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:21.077 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:21.077 INFO:tasks.workunit.client.0.smithi032.stdout:12 2024-09-13T12:19:21.077 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T12:19:21.078 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:21.078 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:19:22.175 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:23.176 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:23.176 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:23.177 INFO:tasks.workunit.client.0.smithi032.stdout:13 2024-09-13T12:19:23.177 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T12:19:23.177 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:23.177 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:19:24.026 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:25.028 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:25.029 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:25.029 INFO:tasks.workunit.client.0.smithi032.stdout:14 2024-09-13T12:19:25.030 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T12:19:25.030 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:25.030 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:19:25.814 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:26.816 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:26.816 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:26.816 INFO:tasks.workunit.client.0.smithi032.stdout:15 2024-09-13T12:19:26.817 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T12:19:26.817 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:26.817 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:19:27.546 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:28.548 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:28.548 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:28.548 INFO:tasks.workunit.client.0.smithi032.stdout:16 2024-09-13T12:19:28.549 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T12:19:28.549 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:28.549 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:19:29.146 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:30.147 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:30.147 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:30.148 INFO:tasks.workunit.client.0.smithi032.stdout:17 2024-09-13T12:19:30.148 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T12:19:30.148 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:30.148 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:19:31.120 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:32.121 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:32.121 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:32.122 INFO:tasks.workunit.client.0.smithi032.stdout:18 2024-09-13T12:19:32.122 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T12:19:32.122 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:32.122 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:19:32.578 INFO:tasks.workunit.client.0.smithi032.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/58499577,v1:127.0.0.1:6819/58499577] [v2:127.0.0.1:6820/58499577,v1:127.0.0.1:6821/58499577] exists,up acb74e48-8aaf-44e5-8f27-a93216246cf7 2024-09-13T12:19:32.581 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:19:32.581 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:19:32.582 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:19:32.582 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2024-09-13T12:19:32.582 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ec-error-rollforward 2024-09-13T12:19:32.582 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T12:19:32.582 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-13T12:19:32.582 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T12:19:32.582 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ec-error-rollforward/3 2024-09-13T12:19:32.582 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7fa1b90f-27d7-414a-8403-1b3eb21a3b85 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-13T12:19:32.582 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:19:32.582 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:19:32.582 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:19:32.582 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-09-13T12:19:32.583 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-09-13T12:19:32.583 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T12:19:32.583 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T12:19:32.583 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-13T12:19:32.583 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T12:19:32.583 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:19:32.583 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:19:32.583 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:19:32.583 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:19:32.583 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90678 2024-09-13T12:19:32.583 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.90678/$cluster-$name.asok' 2024-09-13T12:19:32.583 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90678/$cluster-$name.asok' 2024-09-13T12:19:32.583 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:19:32.584 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T12:19:32.584 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T12:19:32.584 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-13T12:19:32.584 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-13T12:19:32.584 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:19:32.584 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:19:32.584 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:19:32.584 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:19:32.584 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T12:19:32.584 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T12:19:32.584 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ec-error-rollforward/3 2024-09-13T12:19:32.584 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T12:19:32.584 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5e3eebc9-ce73-4d8b-9145-17ce67592096 2024-09-13T12:19:32.584 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 5e3eebc9-ce73-4d8b-9145-17ce67592096' 2024-09-13T12:19:32.585 INFO:tasks.workunit.client.0.smithi032.stdout:add osd3 5e3eebc9-ce73-4d8b-9145-17ce67592096 2024-09-13T12:19:32.585 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T12:19:32.599 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDULeRmQcWoIxAA1l/dZg6lOFhzS/RNXLk7fQ== 2024-09-13T12:19:32.599 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDULeRmQcWoIxAA1l/dZg6lOFhzS/RNXLk7fQ=="}' 2024-09-13T12:19:32.599 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5e3eebc9-ce73-4d8b-9145-17ce67592096 -i td/ec-error-rollforward/3/new.json 2024-09-13T12:19:33.060 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:19:33.071 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ec-error-rollforward/3/new.json 2024-09-13T12:19:33.072 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=7fa1b90f-27d7-414a-8403-1b3eb21a3b85 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90678/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDULeRmQcWoIxAA1l/dZg6lOFhzS/RNXLk7fQ== --osd-uuid 5e3eebc9-ce73-4d8b-9145-17ce67592096 2024-09-13T12:19:33.101 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:19:33.100+0000 7f28c5616640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:19:33.104 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:19:33.103+0000 7f28c5616640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:19:33.105 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:19:33.104+0000 7f28c5616640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:19:33.105 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:19:33.105+0000 7f28c5616640 -1 bdev(0x55dd2a651c00 td/ec-error-rollforward/3/block) open stat got: (1) Operation not permitted 2024-09-13T12:19:33.105 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:19:33.105+0000 7f28c5616640 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2024-09-13T12:19:39.210 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2024-09-13T12:19:39.210 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T12:19:39.211 INFO:tasks.workunit.client.0.smithi032.stdout:adding osd3 key to auth repository 2024-09-13T12:19:39.211 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-13T12:19:39.211 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T12:19:39.779 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.3 2024-09-13T12:19:39.780 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-13T12:19:39.780 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=7fa1b90f-27d7-414a-8403-1b3eb21a3b85 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90678/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:19:39.780 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T12:19:39.780 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T12:19:39.780 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T12:19:39.805 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:19:39.805+0000 7f878401f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:19:39.826 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:19:39.826+0000 7f878401f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:19:39.827 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:19:39.827+0000 7f878401f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:19:40.232 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-13T12:19:40.233 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:19:40.233 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T12:19:40.233 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:19:40.233 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:19:40.233 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:40.233 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:19:40.233 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:19:40.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:40.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:19:40.667 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:41.162 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:19:41.161+0000 7f878401f640 -1 Falling back to public interface 2024-09-13T12:19:41.669 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:41.669 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:41.669 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:19:41.670 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:19:41.670 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:41.670 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:19:42.118 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:43.120 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:43.120 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:43.121 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:19:43.121 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:19:43.121 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:43.121 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:19:43.532 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:43.680 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:19:43.679+0000 7f878401f640 -1 osd.3 0 log_to_monitors true 2024-09-13T12:19:44.533 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:44.533 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:44.533 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:19:44.534 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:19:44.534 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:44.534 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:19:44.949 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:45.950 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:45.951 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:45.951 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:19:45.951 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:19:45.952 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:45.952 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:19:46.364 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:47.365 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:47.365 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:47.365 INFO:tasks.workunit.client.0.smithi032.stdout:5 2024-09-13T12:19:47.366 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T12:19:47.366 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:47.366 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:19:47.883 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:48.884 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:48.884 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:48.885 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T12:19:48.885 INFO:tasks.workunit.client.0.smithi032.stdout:6 2024-09-13T12:19:48.885 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:48.885 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:19:49.300 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:50.301 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:50.301 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:50.301 INFO:tasks.workunit.client.0.smithi032.stdout:7 2024-09-13T12:19:50.302 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T12:19:50.302 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:50.302 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:19:50.805 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:51.806 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:51.807 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:51.807 INFO:tasks.workunit.client.0.smithi032.stdout:8 2024-09-13T12:19:51.807 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T12:19:51.807 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:51.808 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:19:52.237 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:53.238 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:53.239 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:53.239 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T12:19:53.239 INFO:tasks.workunit.client.0.smithi032.stdout:9 2024-09-13T12:19:53.239 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:53.239 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:19:53.743 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:54.745 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:54.745 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:54.745 INFO:tasks.workunit.client.0.smithi032.stdout:10 2024-09-13T12:19:54.746 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T12:19:54.746 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:54.746 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:19:55.187 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:56.189 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:56.189 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:56.189 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T12:19:56.189 INFO:tasks.workunit.client.0.smithi032.stdout:11 2024-09-13T12:19:56.189 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:56.189 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:19:56.598 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:57.599 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:57.599 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:57.600 INFO:tasks.workunit.client.0.smithi032.stdout:12 2024-09-13T12:19:57.600 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T12:19:57.600 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:57.600 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:19:58.081 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:19:59.083 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:19:59.083 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:19:59.083 INFO:tasks.workunit.client.0.smithi032.stdout:13 2024-09-13T12:19:59.083 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T12:19:59.084 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:19:59.084 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:19:59.895 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:20:00.897 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:20:00.897 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:20:00.897 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T12:20:00.897 INFO:tasks.workunit.client.0.smithi032.stdout:14 2024-09-13T12:20:00.897 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:20:00.898 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:20:01.614 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:20:02.616 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:20:02.616 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:20:02.616 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T12:20:02.616 INFO:tasks.workunit.client.0.smithi032.stdout:15 2024-09-13T12:20:02.617 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:20:02.617 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:20:03.655 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:20:04.656 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:20:04.656 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:20:04.657 INFO:tasks.workunit.client.0.smithi032.stdout:16 2024-09-13T12:20:04.657 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T12:20:04.657 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:20:04.657 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:20:05.577 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:20:06.578 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:20:06.578 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:20:06.578 INFO:tasks.workunit.client.0.smithi032.stdout:17 2024-09-13T12:20:06.579 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T12:20:06.579 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:20:06.579 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:20:07.452 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:20:08.453 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:20:08.453 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:20:08.453 INFO:tasks.workunit.client.0.smithi032.stdout:18 2024-09-13T12:20:08.454 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T12:20:08.454 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:20:08.454 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:20:08.898 INFO:tasks.workunit.client.0.smithi032.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2729475811,v1:127.0.0.1:6827/2729475811] [v2:127.0.0.1:6828/2729475811,v1:127.0.0.1:6829/2729475811] exists,up 5e3eebc9-ce73-4d8b-9145-17ce67592096 2024-09-13T12:20:08.899 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:20:08.899 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:20:08.899 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:20:08.899 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2024-09-13T12:20:09.517 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2024-09-13T12:20:10.315 INFO:tasks.workunit.client.0.smithi032.stderr:pool 'ec' already exists 2024-09-13T12:20:10.326 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2024-09-13T12:20:11.797 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2024-09-13T12:20:11.798 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 95289 2024-09-13T12:20:11.798 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=98069 2024-09-13T12:20:11.798 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2024-09-13T12:20:11.798 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2024-09-13T12:20:12.800 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 98090' 2024-09-13T12:20:12.801 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2024-09-13T12:20:12.801 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 98091' 2024-09-13T12:20:12.801 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2024-09-13T12:20:12.801 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 98092' 2024-09-13T12:20:12.801 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2024-09-13T12:20:12.801 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2024-09-13T12:20:13.802 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2024-09-13T12:20:13.803 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:20:13.803 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:20:13.803 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:20:13.803 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:20:13.803 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:20:13.912 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:20:13.912 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 98069 98090 98091 98092 2024-09-13T12:20:13.912 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2024-09-13T12:20:13.917 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2024-09-13T12:20:13.917 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ec-error-rollforward 2024-09-13T12:20:13.917 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:20:13.917 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T12:20:13.917 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:20:13.917 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ec-error-rollforward/0 2024-09-13T12:20:13.917 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=7fa1b90f-27d7-414a-8403-1b3eb21a3b85 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-13T12:20:13.918 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:20:13.918 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:20:13.918 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:20:13.918 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-09-13T12:20:13.918 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-09-13T12:20:13.918 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:20:13.918 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:20:13.918 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-13T12:20:13.918 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:20:13.918 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:20:13.918 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:20:13.918 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:20:13.918 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:20:13.918 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90678 2024-09-13T12:20:13.919 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.90678/$cluster-$name.asok' 2024-09-13T12:20:13.919 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90678/$cluster-$name.asok' 2024-09-13T12:20:13.919 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:20:13.919 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-13T12:20:13.919 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-13T12:20:13.919 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:20:13.919 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:20:13.919 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:20:13.919 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:20:13.919 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:20:13.919 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:20:13.919 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ec-error-rollforward/0 2024-09-13T12:20:13.920 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.0 2024-09-13T12:20:13.920 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T12:20:13.921 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=7fa1b90f-27d7-414a-8403-1b3eb21a3b85 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90678/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:20:13.921 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ec-error-rollforward/0/whoami 2024-09-13T12:20:13.921 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T12:20:13.921 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:20:13.921 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:20:13.921 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:20:13.945 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:13.943+0000 7f077a842640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:20:13.947 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:13.946+0000 7f077a842640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:20:13.949 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:13.947+0000 7f077a842640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:20:14.361 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T12:20:14.361 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:20:14.361 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T12:20:14.361 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:20:14.362 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:20:14.362 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:20:14.362 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:20:14.363 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:20:14.363 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:20:14.363 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:20:14.795 INFO:tasks.workunit.client.0.smithi032.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1038799336,v1:127.0.0.1:6803/1038799336] [v2:127.0.0.1:6804/1038799336,v1:127.0.0.1:6805/1038799336] exists,up 2793142e-b2d9-4bef-88dc-a6fc3289aaaf 2024-09-13T12:20:14.799 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:20:14.799 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:20:14.799 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:20:14.799 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2024-09-13T12:20:14.799 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ec-error-rollforward 2024-09-13T12:20:14.799 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:20:14.799 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T12:20:14.799 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:20:14.799 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ec-error-rollforward/1 2024-09-13T12:20:14.799 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=7fa1b90f-27d7-414a-8403-1b3eb21a3b85 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-13T12:20:14.800 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:20:14.800 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:20:14.800 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:20:14.800 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-09-13T12:20:14.800 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-09-13T12:20:14.800 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:20:14.800 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:20:14.800 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-13T12:20:14.800 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:20:14.800 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:20:14.800 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:20:14.800 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:20:14.801 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:20:14.801 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90678 2024-09-13T12:20:14.801 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.90678/$cluster-$name.asok' 2024-09-13T12:20:14.801 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90678/$cluster-$name.asok' 2024-09-13T12:20:14.801 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:20:14.801 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-13T12:20:14.801 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-13T12:20:14.801 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:20:14.801 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:20:14.801 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:20:14.801 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:20:14.801 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:20:14.802 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:20:14.802 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ec-error-rollforward/1 2024-09-13T12:20:14.802 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.1 2024-09-13T12:20:14.803 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T12:20:14.803 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=7fa1b90f-27d7-414a-8403-1b3eb21a3b85 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90678/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:20:14.803 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ec-error-rollforward/1/whoami 2024-09-13T12:20:14.803 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T12:20:14.803 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:20:14.803 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:20:14.803 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:20:14.823 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:14.821+0000 7f73cb8f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:20:14.825 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:14.823+0000 7f73cb8f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:20:14.826 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:14.825+0000 7f73cb8f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:20:15.231 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T12:20:15.232 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:20:15.232 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T12:20:15.232 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:20:15.232 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:20:15.232 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:20:15.232 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:20:15.232 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:20:15.233 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:20:15.233 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:20:15.292 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:15.291+0000 7f077a842640 -1 Falling back to public interface 2024-09-13T12:20:15.628 INFO:tasks.workunit.client.0.smithi032.stdout:osd.1 up in weight 1 up_from 10 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3503804840,v1:127.0.0.1:6811/3503804840] [v2:127.0.0.1:6812/3503804840,v1:127.0.0.1:6813/3503804840] exists,up ea4118e8-a0ea-4a26-8a3c-16f74116fed1 2024-09-13T12:20:15.631 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:20:15.632 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:20:15.632 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:20:15.632 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2024-09-13T12:20:15.632 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ec-error-rollforward 2024-09-13T12:20:15.632 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:20:15.632 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-13T12:20:15.632 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:20:15.632 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ec-error-rollforward/2 2024-09-13T12:20:15.632 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=7fa1b90f-27d7-414a-8403-1b3eb21a3b85 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-13T12:20:15.632 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:20:15.632 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:20:15.632 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:20:15.632 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-09-13T12:20:15.633 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-09-13T12:20:15.633 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:20:15.633 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:20:15.633 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-13T12:20:15.633 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:20:15.633 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:20:15.633 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:20:15.633 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:20:15.633 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:20:15.633 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90678 2024-09-13T12:20:15.633 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.90678/$cluster-$name.asok' 2024-09-13T12:20:15.633 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90678/$cluster-$name.asok' 2024-09-13T12:20:15.634 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:20:15.634 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-13T12:20:15.634 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-13T12:20:15.634 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:20:15.634 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:20:15.634 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:20:15.634 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:20:15.634 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:20:15.634 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:20:15.634 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ec-error-rollforward/2 2024-09-13T12:20:15.634 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.2 2024-09-13T12:20:15.635 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-13T12:20:15.635 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=7fa1b90f-27d7-414a-8403-1b3eb21a3b85 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90678/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:20:15.635 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ec-error-rollforward/2/whoami 2024-09-13T12:20:15.636 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-13T12:20:15.636 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:20:15.636 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:20:15.636 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:20:15.656 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:15.654+0000 7f1860e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:20:15.658 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:15.657+0000 7f1860e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:20:15.659 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:15.658+0000 7f1860e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:20:15.709 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:15.707+0000 7f73cb8f1640 -1 Falling back to public interface 2024-09-13T12:20:16.075 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-13T12:20:16.075 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:20:16.075 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T12:20:16.075 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:20:16.075 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:20:16.075 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:20:16.076 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:20:16.076 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:20:16.076 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:20:16.076 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:20:16.501 INFO:tasks.workunit.client.0.smithi032.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/58499577,v1:127.0.0.1:6819/58499577] [v2:127.0.0.1:6820/58499577,v1:127.0.0.1:6821/58499577] exists,up acb74e48-8aaf-44e5-8f27-a93216246cf7 2024-09-13T12:20:16.504 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:20:16.504 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:20:16.504 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:20:16.504 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2024-09-13T12:20:16.504 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ec-error-rollforward 2024-09-13T12:20:16.504 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:20:16.504 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-13T12:20:16.504 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:20:16.505 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ec-error-rollforward/3 2024-09-13T12:20:16.505 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=7fa1b90f-27d7-414a-8403-1b3eb21a3b85 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-13T12:20:16.505 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:20:16.505 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:20:16.505 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:20:16.505 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-09-13T12:20:16.505 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-09-13T12:20:16.505 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:20:16.505 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:20:16.505 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-13T12:20:16.505 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:20:16.505 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:20:16.505 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:20:16.506 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:20:16.506 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:20:16.506 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90678 2024-09-13T12:20:16.506 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.90678/$cluster-$name.asok' 2024-09-13T12:20:16.506 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90678/$cluster-$name.asok' 2024-09-13T12:20:16.506 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:20:16.506 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-13T12:20:16.506 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-13T12:20:16.506 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:20:16.506 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:20:16.506 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:20:16.506 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:20:16.506 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:20:16.506 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:20:16.506 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ec-error-rollforward/3 2024-09-13T12:20:16.507 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.3 2024-09-13T12:20:16.508 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-13T12:20:16.508 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=7fa1b90f-27d7-414a-8403-1b3eb21a3b85 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90678/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:20:16.508 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ec-error-rollforward/3/whoami 2024-09-13T12:20:16.508 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-13T12:20:16.508 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:20:16.508 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:20:16.508 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:20:16.528 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:16.527+0000 7fa8d8d56640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:20:16.530 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:16.528+0000 7fa8d8d56640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:20:16.531 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:16.529+0000 7fa8d8d56640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:20:16.764 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:16.763+0000 7f1860e0d640 -1 Falling back to public interface 2024-09-13T12:20:16.971 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-13T12:20:16.971 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:20:16.971 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T12:20:16.971 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:20:16.971 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:20:16.971 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:20:16.971 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:20:16.971 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:20:16.972 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:20:16.972 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:20:17.370 INFO:tasks.workunit.client.0.smithi032.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2729475811,v1:127.0.0.1:6827/2729475811] [v2:127.0.0.1:6828/2729475811,v1:127.0.0.1:6829/2729475811] exists,up 5e3eebc9-ce73-4d8b-9145-17ce67592096 2024-09-13T12:20:17.371 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:20:17.371 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:20:17.371 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:20:17.371 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2024-09-13T12:20:17.371 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T12:20:17.371 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T12:20:17.372 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T12:20:17.372 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T12:20:17.372 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T12:20:17.372 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T12:20:17.372 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T12:20:17.372 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T12:20:17.372 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T12:20:17.458 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T12:20:17.458 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T12:20:17.458 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T12:20:17.459 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T12:20:17.459 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T12:20:17.459 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T12:20:17.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T12:20:17.931 INFO:tasks.workunit.client.0.smithi032.stderr:1 2024-09-13T12:20:17.931 INFO:tasks.workunit.client.0.smithi032.stderr:2 2024-09-13T12:20:17.931 INFO:tasks.workunit.client.0.smithi032.stderr:3' 2024-09-13T12:20:17.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T12:20:17.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:20:17.931 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T12:20:18.409 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:18.408+0000 7fa8d8d56640 -1 Falling back to public interface 2024-09-13T12:20:18.724 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:18.724+0000 7f077a842640 -1 osd.0 26 log_to_monitors true 2024-09-13T12:20:19.571 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:19.570+0000 7f73cb8f1640 -1 osd.1 26 log_to_monitors true 2024-09-13T12:20:21.022 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:21.021+0000 7f1860e0d640 -1 osd.2 25 log_to_monitors true 2024-09-13T12:20:22.294 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:22.293+0000 7fa8d8d56640 -1 osd.3 26 log_to_monitors true 2024-09-13T12:20:23.221 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051587 2024-09-13T12:20:23.221 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051587 2024-09-13T12:20:23.221 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051587' 2024-09-13T12:20:23.221 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:20:23.222 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T12:20:23.353 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=137438953475 2024-09-13T12:20:23.353 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 137438953475 2024-09-13T12:20:23.353 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051587 1-137438953475' 2024-09-13T12:20:23.353 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:20:23.354 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T12:20:23.588 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822659 2024-09-13T12:20:23.588 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822659 2024-09-13T12:20:23.588 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051587 1-137438953475 2-154618822659' 2024-09-13T12:20:23.589 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:20:23.589 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T12:20:23.740 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757251 2024-09-13T12:20:23.740 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757251 2024-09-13T12:20:23.740 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051587 1-137438953475 2-154618822659 3-163208757251' 2024-09-13T12:20:23.740 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:20:23.740 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-124554051587 2024-09-13T12:20:23.740 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:20:23.741 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T12:20:23.742 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-124554051587 2024-09-13T12:20:23.742 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:20:23.743 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051587 2024-09-13T12:20:23.743 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.0 seq 124554051587 2024-09-13T12:20:23.743 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 124554051587' 2024-09-13T12:20:23.743 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:20:24.196 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051587 -lt 124554051587 2024-09-13T12:20:24.196 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:20:24.196 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-137438953475 2024-09-13T12:20:24.196 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:20:24.197 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T12:20:24.198 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-137438953475 2024-09-13T12:20:24.198 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:20:24.199 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=137438953475 2024-09-13T12:20:24.200 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.1 seq 137438953475 2024-09-13T12:20:24.200 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 137438953475' 2024-09-13T12:20:24.200 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T12:20:24.678 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 137438953475 -lt 137438953475 2024-09-13T12:20:24.679 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:20:24.679 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-154618822659 2024-09-13T12:20:24.679 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:20:24.680 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T12:20:24.681 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-154618822659 2024-09-13T12:20:24.681 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:20:24.682 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822659 2024-09-13T12:20:24.682 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.2 seq 154618822659 2024-09-13T12:20:24.682 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 154618822659' 2024-09-13T12:20:24.682 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T12:20:25.130 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822659 -lt 154618822659 2024-09-13T12:20:25.130 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:20:25.130 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-163208757251 2024-09-13T12:20:25.130 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:20:25.131 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T12:20:25.132 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-163208757251 2024-09-13T12:20:25.132 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:20:25.133 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757251 2024-09-13T12:20:25.133 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 163208757251' 2024-09-13T12:20:25.133 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.3 seq 163208757251 2024-09-13T12:20:25.134 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T12:20:25.538 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757251 -lt 163208757251 2024-09-13T12:20:25.538 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T12:20:25.539 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:20:25.539 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:20:26.086 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T12:20:26.086 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:20:26.086 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:20:26.086 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:20:26.086 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:20:26.086 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:20:26.087 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:20:26.087 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:20:26.509 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T12:20:26.510 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:20:26.510 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:20:26.510 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:20:27.051 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T12:20:27.051 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T12:20:27.051 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T12:20:27.051 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2024-09-13T12:20:27.051 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ec-error-rollforward 2024-09-13T12:20:27.052 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T12:20:27.052 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ec-error-rollforward KILL 2024-09-13T12:20:27.052 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:20:27.052 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:20:27.052 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:20:27.052 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:20:27.052 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:20:27.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:20:27.160 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T12:20:27.161 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T12:20:27.161 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T12:20:27.162 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T12:20:27.162 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T12:20:27.162 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T12:20:27.163 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:20:27.163 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T12:20:27.164 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T12:20:27.164 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:20:27.164 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:20:27.165 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T12:20:27.165 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ec-error-rollforward 2024-09-13T12:20:27.183 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T12:20:27.183 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:20:27.183 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90678 2024-09-13T12:20:27.183 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.90678 2024-09-13T12:20:27.185 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T12:20:27.185 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T12:20:27.185 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-13T12:20:27.185 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/ec-error-rollforward 0 2024-09-13T12:20:27.185 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ec-error-rollforward 2024-09-13T12:20:27.185 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-13T12:20:27.185 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ec-error-rollforward KILL 2024-09-13T12:20:27.185 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:20:27.185 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:20:27.185 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:20:27.185 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:20:27.186 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:20:27.186 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:20:27.186 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T12:20:27.187 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T12:20:27.187 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T12:20:27.188 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T12:20:27.189 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T12:20:27.189 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T12:20:27.189 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:20:27.190 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T12:20:27.190 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T12:20:27.190 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:20:27.190 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:20:27.191 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-13T12:20:27.192 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ec-error-rollforward 2024-09-13T12:20:27.194 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T12:20:27.194 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:20:27.195 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90678 2024-09-13T12:20:27.195 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.90678 2024-09-13T12:20:27.195 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T12:20:27.195 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T12:20:27.195 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-13T12:20:27.195 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-13T12:20:27.195 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-13T12:20:27.260 INFO:tasks.workunit:Running workunit osd/osd-bench.sh... 2024-09-13T12:20:27.261 DEBUG:teuthology.orchestra.run.smithi032:workunit test osd/osd-bench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d58f73069a8157a9ace12a0247ee3e8497707ea9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh 2024-09-13T12:20:27.323 INFO:tasks.workunit.client.0.smithi032.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-13T12:20:27.328 INFO:tasks.workunit.client.0.smithi032.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-13T12:20:27.329 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-13T12:20:27.329 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-13T12:20:27.329 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-13T12:20:27.329 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-13T12:20:27.329 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-13T12:20:27.329 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-13T12:20:27.329 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-13T12:20:27.329 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-13T12:20:27.329 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-bench 2024-09-13T12:20:27.329 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:22: run: local dir=td/osd-bench 2024-09-13T12:20:27.329 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:23: run: shift 2024-09-13T12:20:27.329 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: export CEPH_MON=127.0.0.1:7106 2024-09-13T12:20:27.329 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: CEPH_MON=127.0.0.1:7106 2024-09-13T12:20:27.329 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:26: run: export CEPH_ARGS 2024-09-13T12:20:27.330 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: uuidgen 2024-09-13T12:20:27.330 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: CEPH_ARGS+='--fsid=ec340106-8452-4be7-8cb5-5b1432a06ff3 --auth-supported=none ' 2024-09-13T12:20:27.330 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7106 ' 2024-09-13T12:20:27.330 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:29: run: CEPH_ARGS+='--debug-bluestore 20 ' 2024-09-13T12:20:27.331 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: set 2024-09-13T12:20:27.331 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-13T12:20:27.331 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: local funcs=TEST_bench 2024-09-13T12:20:27.331 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:32: run: for func in $funcs 2024-09-13T12:20:27.332 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:33: run: setup td/osd-bench 2024-09-13T12:20:27.332 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-bench 2024-09-13T12:20:27.332 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-bench 2024-09-13T12:20:27.332 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bench 2024-09-13T12:20:27.332 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T12:20:27.332 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bench KILL 2024-09-13T12:20:27.332 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:20:27.332 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:20:27.332 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:20:27.332 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:20:27.332 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:20:27.333 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:20:27.333 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T12:20:27.333 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T12:20:27.334 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T12:20:27.335 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T12:20:27.335 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T12:20:27.335 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T12:20:27.336 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:20:27.336 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T12:20:27.336 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T12:20:27.336 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:20:27.337 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:20:27.338 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T12:20:27.338 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bench 2024-09-13T12:20:27.339 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T12:20:27.339 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:20:27.339 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.100637 2024-09-13T12:20:27.339 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.100637 2024-09-13T12:20:27.340 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T12:20:27.340 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T12:20:27.340 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-bench 2024-09-13T12:20:27.341 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T12:20:27.341 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:20:27.341 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.100637 2024-09-13T12:20:27.342 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.100637 2024-09-13T12:20:27.343 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T12:20:27.344 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-13T12:20:27.344 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-13T12:20:27.344 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T12:20:27.344 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-bench 1' TERM HUP INT 2024-09-13T12:20:27.344 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:34: run: TEST_bench td/osd-bench 2024-09-13T12:20:27.344 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:40: TEST_bench: local dir=td/osd-bench 2024-09-13T12:20:27.344 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:42: TEST_bench: run_mon td/osd-bench a 2024-09-13T12:20:27.344 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-bench 2024-09-13T12:20:27.344 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T12:20:27.344 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T12:20:27.344 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T12:20:27.344 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-bench/a 2024-09-13T12:20:27.344 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bench/a --run-dir=td/osd-bench 2024-09-13T12:20:27.822 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T12:20:27.822 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:20:27.822 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:20:27.822 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:20:27.822 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:20:27.822 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.100637 2024-09-13T12:20:27.822 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.100637/$cluster-$name.asok' 2024-09-13T12:20:27.822 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bench/a '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.100637/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bench/log --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-13T12:20:28.006 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T12:20:28.007 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T12:20:28.007 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T12:20:28.007 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T12:20:28.007 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T12:20:28.007 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T12:20:28.008 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T12:20:28.008 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T12:20:28.008 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T12:20:28.009 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:20:28.009 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:20:28.009 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.100637 2024-09-13T12:20:28.009 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.100637/ceph-mon.a.asok 2024-09-13T12:20:28.009 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T12:20:28.009 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.100637/ceph-mon.a.asok config get fsid 2024-09-13T12:20:28.095 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T12:20:28.096 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T12:20:28.096 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T12:20:28.096 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T12:20:28.096 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T12:20:28.096 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T12:20:28.096 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T12:20:28.096 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T12:20:28.096 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:20:28.096 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:20:28.096 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.100637 2024-09-13T12:20:28.096 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.100637/ceph-mon.a.asok 2024-09-13T12:20:28.096 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T12:20:28.096 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.100637/ceph-mon.a.asok config get mon_host 2024-09-13T12:20:28.177 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:43: TEST_bench: run_mgr td/osd-bench x 2024-09-13T12:20:28.177 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-bench 2024-09-13T12:20:28.177 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T12:20:28.177 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T12:20:28.177 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T12:20:28.177 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-bench/x 2024-09-13T12:20:28.177 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T12:20:29.028 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T12:20:29.029 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:20:29.029 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:20:29.029 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:20:29.029 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:20:29.029 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.100637 2024-09-13T12:20:29.029 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.100637/$cluster-$name.asok' 2024-09-13T12:20:29.029 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T12:20:29.029 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bench/x '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.100637/$cluster-$name.asok' --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T12:20:29.060 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:44: TEST_bench: run_osd td/osd-bench 0 2024-09-13T12:20:29.060 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bench 2024-09-13T12:20:29.060 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T12:20:29.060 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T12:20:29.060 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T12:20:29.060 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bench/0 2024-09-13T12:20:29.060 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ec340106-8452-4be7-8cb5-5b1432a06ff3 --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 ' 2024-09-13T12:20:29.060 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:20:29.060 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:20:29.061 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:20:29.061 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bench/0' 2024-09-13T12:20:29.061 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bench/0/journal' 2024-09-13T12:20:29.061 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T12:20:29.061 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T12:20:29.061 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bench' 2024-09-13T12:20:29.061 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T12:20:29.061 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:20:29.061 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:20:29.061 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:20:29.061 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:20:29.061 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.100637 2024-09-13T12:20:29.061 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.100637/$cluster-$name.asok' 2024-09-13T12:20:29.061 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.100637/$cluster-$name.asok' 2024-09-13T12:20:29.062 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:20:29.062 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T12:20:29.062 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T12:20:29.062 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bench/$name.log' 2024-09-13T12:20:29.062 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bench/$name.pid' 2024-09-13T12:20:29.062 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:20:29.062 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:20:29.062 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:20:29.062 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:20:29.062 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T12:20:29.062 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T12:20:29.062 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bench/0 2024-09-13T12:20:29.062 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T12:20:29.063 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=79e0883e-304d-4cf9-ab79-f4f258acb2db 2024-09-13T12:20:29.063 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 79e0883e-304d-4cf9-ab79-f4f258acb2db' 2024-09-13T12:20:29.063 INFO:tasks.workunit.client.0.smithi032.stdout:add osd0 79e0883e-304d-4cf9-ab79-f4f258acb2db 2024-09-13T12:20:29.063 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T12:20:29.078 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQANLuRmboWdBBAA0wLC6/KHCCQyw3DPSKOKww== 2024-09-13T12:20:29.078 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQANLuRmboWdBBAA0wLC6/KHCCQyw3DPSKOKww=="}' 2024-09-13T12:20:29.078 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 79e0883e-304d-4cf9-ab79-f4f258acb2db -i td/osd-bench/0/new.json 2024-09-13T12:20:29.356 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:20:29.366 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bench/0/new.json 2024-09-13T12:20:29.366 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ec340106-8452-4be7-8cb5-5b1432a06ff3 --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.100637/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQANLuRmboWdBBAA0wLC6/KHCCQyw3DPSKOKww== --osd-uuid 79e0883e-304d-4cf9-ab79-f4f258acb2db 2024-09-13T12:20:29.395 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:29.394+0000 7f0ddaae0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:20:29.423 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:29.423+0000 7f0ddaae0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:20:29.424 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:29.424+0000 7f0ddaae0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:20:29.425 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:29.424+0000 7f0ddaae0640 -1 bdev(0x561a3e8c9c00 td/osd-bench/0/block) open stat got: (1) Operation not permitted 2024-09-13T12:20:29.425 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:29.424+0000 7f0ddaae0640 -1 bluestore(td/osd-bench/0) _read_fsid unparsable uuid 2024-09-13T12:20:35.141 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bench/0/keyring 2024-09-13T12:20:35.141 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T12:20:35.142 INFO:tasks.workunit.client.0.smithi032.stdout:adding osd0 key to auth repository 2024-09-13T12:20:35.142 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T12:20:35.142 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bench/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T12:20:35.703 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.0 2024-09-13T12:20:35.704 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T12:20:35.704 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ec340106-8452-4be7-8cb5-5b1432a06ff3 --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.100637/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:20:35.704 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T12:20:35.704 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T12:20:35.704 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T12:20:35.729 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:35.728+0000 7fd34d9c1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:20:35.753 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:35.752+0000 7fd34d9c1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:20:35.754 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:35.753+0000 7fd34d9c1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:20:36.183 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T12:20:36.183 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:20:36.183 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T12:20:36.183 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:20:36.183 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:20:36.183 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:20:36.184 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:20:36.184 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:20:36.184 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:20:36.184 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:20:36.592 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:36.591+0000 7fd34d9c1640 -1 Falling back to public interface 2024-09-13T12:20:36.615 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:20:37.616 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:20:37.617 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:20:37.617 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:20:37.617 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:20:37.618 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:20:37.618 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:20:38.060 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:20:39.061 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:20:39.062 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:20:39.062 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:20:39.062 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:20:39.062 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:20:39.062 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:20:39.473 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:20:39.613 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:20:39.612+0000 7fd34d9c1640 -1 osd.0 0 log_to_monitors true 2024-09-13T12:20:40.474 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:20:40.474 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:20:40.475 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:20:40.475 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:20:40.475 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:20:40.475 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:20:40.983 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:20:41.988 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:20:41.988 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:20:41.989 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:20:41.989 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:20:41.989 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:20:41.989 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:20:42.458 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:20:43.459 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:20:43.459 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:20:43.460 INFO:tasks.workunit.client.0.smithi032.stdout:5 2024-09-13T12:20:43.460 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T12:20:43.460 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:20:43.460 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:20:43.887 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:20:44.889 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:20:44.889 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:20:44.889 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T12:20:44.890 INFO:tasks.workunit.client.0.smithi032.stdout:6 2024-09-13T12:20:44.890 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:20:44.890 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:20:45.299 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:20:46.301 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:20:46.301 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:20:46.301 INFO:tasks.workunit.client.0.smithi032.stdout:7 2024-09-13T12:20:46.302 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T12:20:46.302 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:20:46.302 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:20:46.808 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:20:47.809 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:20:47.809 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:20:47.810 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T12:20:47.810 INFO:tasks.workunit.client.0.smithi032.stdout:8 2024-09-13T12:20:47.810 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:20:47.810 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:20:48.226 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:20:49.227 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:20:49.227 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:20:49.227 INFO:tasks.workunit.client.0.smithi032.stdout:9 2024-09-13T12:20:49.228 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T12:20:49.228 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:20:49.228 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:20:49.796 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:20:50.797 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:20:50.797 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:20:50.797 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T12:20:50.797 INFO:tasks.workunit.client.0.smithi032.stdout:10 2024-09-13T12:20:50.798 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:20:50.798 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:20:51.215 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:20:52.217 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:20:52.217 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:20:52.217 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T12:20:52.217 INFO:tasks.workunit.client.0.smithi032.stdout:11 2024-09-13T12:20:52.218 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:20:52.218 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:20:53.131 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:20:54.132 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:20:54.132 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:20:54.133 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T12:20:54.133 INFO:tasks.workunit.client.0.smithi032.stdout:12 2024-09-13T12:20:54.133 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:20:54.133 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:20:54.975 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:20:55.977 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:20:55.977 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:20:55.977 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T12:20:55.977 INFO:tasks.workunit.client.0.smithi032.stdout:13 2024-09-13T12:20:55.978 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:20:55.978 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:20:56.978 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:20:57.979 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:20:57.979 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:20:57.979 INFO:tasks.workunit.client.0.smithi032.stdout:14 2024-09-13T12:20:57.980 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T12:20:57.980 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:20:57.980 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:20:58.794 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:20:59.795 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:20:59.795 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:20:59.795 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T12:20:59.796 INFO:tasks.workunit.client.0.smithi032.stdout:15 2024-09-13T12:20:59.796 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:20:59.796 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:21:00.559 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:21:01.560 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:21:01.561 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:21:01.561 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T12:21:01.561 INFO:tasks.workunit.client.0.smithi032.stdout:16 2024-09-13T12:21:01.561 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:21:01.561 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:21:02.468 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:21:03.472 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:21:03.472 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:21:03.473 INFO:tasks.workunit.client.0.smithi032.stdout:17 2024-09-13T12:21:03.473 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T12:21:03.473 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:21:03.473 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:21:03.905 INFO:tasks.workunit.client.0.smithi032.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3564578369,v1:127.0.0.1:6803/3564578369] [v2:127.0.0.1:6804/3564578369,v1:127.0.0.1:6805/3564578369] exists,up 79e0883e-304d-4cf9-ab79-f4f258acb2db 2024-09-13T12:21:03.905 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:21:03.906 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:21:03.906 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:21:03.906 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: CEPH_ARGS= 2024-09-13T12:21:03.906 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: ceph-conf --show-config-value osd_bench_small_size_max_iops 2024-09-13T12:21:03.920 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: local osd_bench_small_size_max_iops=100 2024-09-13T12:21:03.920 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: CEPH_ARGS= 2024-09-13T12:21:03.920 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: ceph-conf --show-config-value osd_bench_large_size_max_throughput 2024-09-13T12:21:03.934 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: local osd_bench_large_size_max_throughput=104857600 2024-09-13T12:21:03.935 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: CEPH_ARGS= 2024-09-13T12:21:03.935 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: ceph-conf --show-config-value osd_bench_max_block_size 2024-09-13T12:21:03.949 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: local osd_bench_max_block_size=67108864 2024-09-13T12:21:03.949 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: CEPH_ARGS= 2024-09-13T12:21:03.950 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: ceph-conf --show-config-value osd_bench_duration 2024-09-13T12:21:03.964 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: local osd_bench_duration=30 2024-09-13T12:21:03.964 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:58: TEST_bench: expect_failure td/osd-bench osd_bench_max_block_size ceph tell osd.0 bench 1024 67108865 2024-09-13T12:21:03.965 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: expect_failure: local dir=td/osd-bench 2024-09-13T12:21:03.965 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: shift 2024-09-13T12:21:03.965 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: expect_failure: local expected=osd_bench_max_block_size 2024-09-13T12:21:03.965 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: expect_failure: shift 2024-09-13T12:21:03.965 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: local success 2024-09-13T12:21:03.965 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: ceph tell osd.0 bench 1024 67108865 2024-09-13T12:21:04.062 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: success=false 2024-09-13T12:21:04.062 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: false 2024-09-13T12:21:04.062 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: grep --quiet osd_bench_max_block_size td/osd-bench/out 2024-09-13T12:21:04.063 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: expect_failure: return 0 2024-09-13T12:21:04.064 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:64: TEST_bench: local bsize=1024 2024-09-13T12:21:04.064 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:65: TEST_bench: local max_count=3072000 2024-09-13T12:21:04.064 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:66: TEST_bench: expect_failure td/osd-bench bench_small_size_max_iops ceph tell osd.0 bench 3072001 1024 2024-09-13T12:21:04.064 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: expect_failure: local dir=td/osd-bench 2024-09-13T12:21:04.064 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: shift 2024-09-13T12:21:04.064 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: expect_failure: local expected=bench_small_size_max_iops 2024-09-13T12:21:04.064 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: expect_failure: shift 2024-09-13T12:21:04.064 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: local success 2024-09-13T12:21:04.064 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: ceph tell osd.0 bench 3072001 1024 2024-09-13T12:21:04.183 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: success=false 2024-09-13T12:21:04.183 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: false 2024-09-13T12:21:04.183 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: grep --quiet bench_small_size_max_iops td/osd-bench/out 2024-09-13T12:21:04.184 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: expect_failure: return 0 2024-09-13T12:21:04.184 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:72: TEST_bench: local bsize=1048577 2024-09-13T12:21:04.184 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:73: TEST_bench: local max_count=3145728000 2024-09-13T12:21:04.185 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:74: TEST_bench: expect_failure td/osd-bench osd_bench_large_size_max_throughput ceph tell osd.0 bench 3145728001 1048577 2024-09-13T12:21:04.185 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: expect_failure: local dir=td/osd-bench 2024-09-13T12:21:04.185 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: shift 2024-09-13T12:21:04.185 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: expect_failure: local expected=osd_bench_large_size_max_throughput 2024-09-13T12:21:04.185 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: expect_failure: shift 2024-09-13T12:21:04.185 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: local success 2024-09-13T12:21:04.185 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: ceph tell osd.0 bench 3145728001 1048577 2024-09-13T12:21:04.280 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: success=false 2024-09-13T12:21:04.280 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: false 2024-09-13T12:21:04.281 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: grep --quiet osd_bench_large_size_max_throughput td/osd-bench/out 2024-09-13T12:21:04.281 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: expect_failure: return 0 2024-09-13T12:21:04.281 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:80: TEST_bench: ceph tell osd.0 bench 2024-09-13T12:21:55.306 INFO:tasks.workunit.client.0.smithi032.stdout:{ 2024-09-13T12:21:55.307 INFO:tasks.workunit.client.0.smithi032.stdout: "bytes_written": 1073741824, 2024-09-13T12:21:55.307 INFO:tasks.workunit.client.0.smithi032.stdout: "blocksize": 4194304, 2024-09-13T12:21:55.307 INFO:tasks.workunit.client.0.smithi032.stdout: "elapsed_sec": 25.397331621999999, 2024-09-13T12:21:55.307 INFO:tasks.workunit.client.0.smithi032.stdout: "bytes_per_sec": 42277741.614000492, 2024-09-13T12:21:55.307 INFO:tasks.workunit.client.0.smithi032.stdout: "iops": 10.079799083232997 2024-09-13T12:21:55.307 INFO:tasks.workunit.client.0.smithi032.stdout:} 2024-09-13T12:21:55.315 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:84: TEST_bench: ceph tell osd.0 bench 10 14456 4444 3 2024-09-13T12:21:55.712 INFO:tasks.workunit.client.0.smithi032.stdout:{ 2024-09-13T12:21:55.712 INFO:tasks.workunit.client.0.smithi032.stdout: "bytes_written": 10, 2024-09-13T12:21:55.712 INFO:tasks.workunit.client.0.smithi032.stdout: "blocksize": 14456, 2024-09-13T12:21:55.712 INFO:tasks.workunit.client.0.smithi032.stdout: "elapsed_sec": 0.033479786999999997, 2024-09-13T12:21:55.712 INFO:tasks.workunit.client.0.smithi032.stdout: "bytes_per_sec": 298.68768280992947, 2024-09-13T12:21:55.712 INFO:tasks.workunit.client.0.smithi032.stdout: "iops": 0.020661848561837955 2024-09-13T12:21:55.712 INFO:tasks.workunit.client.0.smithi032.stdout:} 2024-09-13T12:21:55.721 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:90: TEST_bench: ceph tell osd.0 bench 1 14456 2024-09-13T12:21:55.971 INFO:tasks.workunit.client.0.smithi032.stdout:{ 2024-09-13T12:21:55.971 INFO:tasks.workunit.client.0.smithi032.stdout: "bytes_written": 1, 2024-09-13T12:21:55.971 INFO:tasks.workunit.client.0.smithi032.stdout: "blocksize": 14456, 2024-09-13T12:21:55.971 INFO:tasks.workunit.client.0.smithi032.stdout: "elapsed_sec": 0.034355705, 2024-09-13T12:21:55.971 INFO:tasks.workunit.client.0.smithi032.stdout: "bytes_per_sec": 29.107247253403767, 2024-09-13T12:21:55.971 INFO:tasks.workunit.client.0.smithi032.stdout: "iops": 0.0020135063124933428 2024-09-13T12:21:55.971 INFO:tasks.workunit.client.0.smithi032.stdout:} 2024-09-13T12:21:55.981 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:35: run: teardown td/osd-bench 2024-09-13T12:21:55.981 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bench 2024-09-13T12:21:55.981 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T12:21:55.981 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bench KILL 2024-09-13T12:21:55.981 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:21:55.981 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:21:55.981 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:21:55.982 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:21:55.982 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:21:56.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:21:56.087 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T12:21:56.088 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T12:21:56.088 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T12:21:56.089 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T12:21:56.089 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T12:21:56.089 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T12:21:56.090 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:21:56.090 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T12:21:56.091 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T12:21:56.091 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:21:56.091 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:21:56.092 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T12:21:56.092 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bench 2024-09-13T12:21:56.118 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T12:21:56.119 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:21:56.119 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.100637 2024-09-13T12:21:56.119 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.100637 2024-09-13T12:21:56.120 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T12:21:56.120 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T12:21:56.120 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-13T12:21:56.120 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-bench 0 2024-09-13T12:21:56.121 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bench 2024-09-13T12:21:56.121 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-13T12:21:56.121 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bench KILL 2024-09-13T12:21:56.121 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:21:56.121 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:21:56.121 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:21:56.121 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:21:56.121 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:21:56.122 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:21:56.122 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T12:21:56.123 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T12:21:56.123 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T12:21:56.124 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T12:21:56.124 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T12:21:56.124 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T12:21:56.125 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:21:56.125 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T12:21:56.125 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T12:21:56.126 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:21:56.126 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:21:56.127 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-13T12:21:56.127 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bench 2024-09-13T12:21:56.128 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T12:21:56.128 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:21:56.129 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.100637 2024-09-13T12:21:56.129 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.100637 2024-09-13T12:21:56.130 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T12:21:56.130 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T12:21:56.130 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-13T12:21:56.131 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-13T12:21:56.131 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-13T12:21:56.196 INFO:tasks.workunit:Running workunit osd/osd-bluefs-volume-ops.sh... 2024-09-13T12:21:56.197 DEBUG:teuthology.orchestra.run.smithi032:workunit test osd/osd-bluefs-volume-ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d58f73069a8157a9ace12a0247ee3e8497707ea9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh 2024-09-13T12:21:56.259 INFO:tasks.workunit.client.0.smithi032.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-13T12:21:56.265 INFO:tasks.workunit.client.0.smithi032.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-13T12:21:56.265 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-13T12:21:56.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-13T12:21:56.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-13T12:21:56.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-13T12:21:56.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-13T12:21:56.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-13T12:21:56.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-13T12:21:56.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-13T12:21:56.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-bluefs-volume-ops 2024-09-13T12:21:56.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:8: run: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:21:56.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:9: run: shift 2024-09-13T12:21:56.266 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: set 2024-09-13T12:21:56.266 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-13T12:21:56.268 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: local 'funcs=TEST_bluestore 2024-09-13T12:21:56.268 INFO:tasks.workunit.client.0.smithi032.stderr:TEST_bluestore2 2024-09-13T12:21:56.268 INFO:tasks.workunit.client.0.smithi032.stderr:TEST_bluestore_expand' 2024-09-13T12:21:56.268 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-09-13T12:21:56.268 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-09-13T12:21:56.268 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:21:56.268 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-bluefs-volume-ops 2024-09-13T12:21:56.268 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:21:56.269 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T12:21:56.269 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-13T12:21:56.269 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:21:56.269 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:21:56.269 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:21:56.269 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:21:56.269 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:21:56.269 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:21:56.269 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T12:21:56.270 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T12:21:56.270 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T12:21:56.271 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T12:21:56.271 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T12:21:56.271 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T12:21:56.272 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:21:56.272 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T12:21:56.273 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T12:21:56.273 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:21:56.273 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:21:56.274 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T12:21:56.274 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-13T12:21:56.276 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T12:21:56.276 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:21:56.276 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:21:56.276 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.102856 2024-09-13T12:21:56.276 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T12:21:56.276 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T12:21:56.277 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-bluefs-volume-ops 2024-09-13T12:21:56.278 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T12:21:56.278 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:21:56.278 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:21:56.278 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.102856 2024-09-13T12:21:56.279 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T12:21:56.279 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-13T12:21:56.279 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-13T12:21:56.279 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T12:21:56.279 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-09-13T12:21:56.280 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore td/osd-bluefs-volume-ops 2024-09-13T12:21:56.280 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:20: TEST_bluestore: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:21:56.281 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: ulimit -n 2024-09-13T12:21:56.281 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: local flimit=4096 2024-09-13T12:21:56.282 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:23: TEST_bluestore: '[' 4096 -lt 1536 ']' 2024-09-13T12:21:56.282 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: export CEPH_MON=127.0.0.1:7146 2024-09-13T12:21:56.282 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: CEPH_MON=127.0.0.1:7146 2024-09-13T12:21:56.282 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:27: TEST_bluestore: export CEPH_ARGS 2024-09-13T12:21:56.282 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: uuidgen 2024-09-13T12:21:56.282 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: CEPH_ARGS+='--fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none ' 2024-09-13T12:21:56.282 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:29: TEST_bluestore: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-13T12:21:56.282 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:30: TEST_bluestore: CEPH_ARGS+='--bluestore_block_size=2147483648 ' 2024-09-13T12:21:56.282 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:31: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-09-13T12:21:56.282 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:32: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-09-13T12:21:56.282 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:33: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_size=536870912 ' 2024-09-13T12:21:56.282 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:34: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_create=true ' 2024-09-13T12:21:56.282 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:35: TEST_bluestore: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-09-13T12:21:56.283 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:37: TEST_bluestore: run_mon td/osd-bluefs-volume-ops a 2024-09-13T12:21:56.283 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:21:56.283 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T12:21:56.283 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T12:21:56.283 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T12:21:56.283 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-09-13T12:21:56.283 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-09-13T12:21:56.687 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T12:21:56.688 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:21:56.688 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:21:56.688 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:21:56.688 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:21:56.688 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:21:56.688 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:21:56.688 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-13T12:21:56.872 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T12:21:56.874 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T12:21:56.874 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T12:21:56.874 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T12:21:56.874 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T12:21:56.874 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T12:21:56.874 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T12:21:56.874 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T12:21:56.874 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T12:21:56.875 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:21:56.875 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:21:56.875 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:21:56.875 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.102856/ceph-mon.a.asok 2024-09-13T12:21:56.875 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T12:21:56.875 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.102856/ceph-mon.a.asok config get fsid 2024-09-13T12:21:56.961 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T12:21:56.961 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T12:21:56.961 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T12:21:56.961 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T12:21:56.961 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T12:21:56.961 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T12:21:56.961 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T12:21:56.961 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T12:21:56.961 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:21:56.961 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:21:56.962 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:21:56.962 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.102856/ceph-mon.a.asok 2024-09-13T12:21:56.962 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T12:21:56.962 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.102856/ceph-mon.a.asok config get mon_host 2024-09-13T12:21:57.042 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:38: TEST_bluestore: run_mgr td/osd-bluefs-volume-ops x 2024-09-13T12:21:57.042 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:21:57.042 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T12:21:57.042 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T12:21:57.042 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T12:21:57.042 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-09-13T12:21:57.042 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T12:21:57.896 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T12:21:57.896 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:21:57.896 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:21:57.897 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:21:57.897 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:21:57.897 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:21:57.897 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:21:57.897 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T12:21:57.897 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T12:21:57.928 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:39: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 0 2024-09-13T12:21:57.928 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:21:57.928 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T12:21:57.928 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T12:21:57.929 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T12:21:57.929 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-13T12:21:57.929 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-13T12:21:57.929 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:21:57.929 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:21:57.929 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:21:57.929 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-13T12:21:57.929 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-13T12:21:57.929 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T12:21:57.929 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T12:21:57.929 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-13T12:21:57.929 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T12:21:57.929 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:21:57.930 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:21:57.930 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:21:57.930 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:21:57.930 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:21:57.930 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:21:57.930 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:21:57.930 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:21:57.930 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T12:21:57.930 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T12:21:57.930 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-13T12:21:57.930 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-13T12:21:57.930 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:21:57.930 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:21:57.930 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:21:57.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:21:57.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T12:21:57.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T12:21:57.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-13T12:21:57.931 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T12:21:57.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f0200d83-d430-4fd5-a4ea-2304279eaa4f 2024-09-13T12:21:57.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 f0200d83-d430-4fd5-a4ea-2304279eaa4f' 2024-09-13T12:21:57.931 INFO:tasks.workunit.client.0.smithi032.stdout:add osd0 f0200d83-d430-4fd5-a4ea-2304279eaa4f 2024-09-13T12:21:57.932 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T12:21:57.947 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBlLuRm6ENdOBAAqNctrR75o3cAXCd0Ev07Lw== 2024-09-13T12:21:57.947 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBlLuRm6ENdOBAAqNctrR75o3cAXCd0Ev07Lw=="}' 2024-09-13T12:21:57.947 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f0200d83-d430-4fd5-a4ea-2304279eaa4f -i td/osd-bluefs-volume-ops/0/new.json 2024-09-13T12:21:58.222 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:21:58.232 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-09-13T12:21:58.233 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBlLuRm6ENdOBAAqNctrR75o3cAXCd0Ev07Lw== --osd-uuid f0200d83-d430-4fd5-a4ea-2304279eaa4f 2024-09-13T12:21:58.261 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:21:58.260+0000 7fe213758640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:21:58.290 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:21:58.289+0000 7fe213758640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:21:58.290 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:21:58.290+0000 7fe213758640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:21:58.291 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:21:58.290+0000 7fe213758640 -1 bdev(0x55a1a77bfc00 td/osd-bluefs-volume-ops/0/block) open stat got: (1) Operation not permitted 2024-09-13T12:21:58.291 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:21:58.291+0000 7fe213758640 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-09-13T12:22:07.466 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-09-13T12:22:07.467 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T12:22:07.467 INFO:tasks.workunit.client.0.smithi032.stdout:adding osd0 key to auth repository 2024-09-13T12:22:07.468 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T12:22:07.468 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T12:22:08.028 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.0 2024-09-13T12:22:08.029 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T12:22:08.029 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:22:08.030 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T12:22:08.030 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T12:22:08.030 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T12:22:08.055 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:22:08.054+0000 7fe36e4e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:22:08.065 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:22:08.065+0000 7fe36e4e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:22:08.066 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:22:08.066+0000 7fe36e4e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:22:08.479 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T12:22:08.479 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:22:08.479 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T12:22:08.479 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:22:08.479 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:22:08.479 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:22:08.480 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:22:08.480 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:22:08.480 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:22:08.480 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:22:08.922 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:22:09.915 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:22:09.914+0000 7fe36e4e2640 -1 Falling back to public interface 2024-09-13T12:22:09.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:22:09.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:22:09.923 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:22:09.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:22:09.924 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:22:09.924 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:22:10.364 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:22:11.366 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:22:11.366 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:22:11.366 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:22:11.366 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:22:11.367 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:22:11.367 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:22:11.778 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:22:12.779 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:22:12.779 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:22:12.779 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:22:12.780 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:22:12.780 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:22:12.780 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:22:13.216 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:22:14.218 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:22:14.218 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:22:14.218 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:22:14.218 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:22:14.218 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:22:14.219 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:22:14.667 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:22:15.146 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:22:15.145+0000 7fe36e4e2640 -1 osd.0 0 log_to_monitors true 2024-09-13T12:22:15.669 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:22:15.669 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:22:15.669 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T12:22:15.669 INFO:tasks.workunit.client.0.smithi032.stdout:5 2024-09-13T12:22:15.670 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:22:15.670 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:22:16.214 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:22:17.215 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:22:17.216 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:22:17.216 INFO:tasks.workunit.client.0.smithi032.stdout:6 2024-09-13T12:22:17.216 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T12:22:17.216 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:22:17.217 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:22:17.635 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:22:18.636 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:22:18.636 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:22:18.636 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T12:22:18.637 INFO:tasks.workunit.client.0.smithi032.stdout:7 2024-09-13T12:22:18.637 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:22:18.637 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:22:19.069 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:22:20.070 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:22:20.070 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:22:20.070 INFO:tasks.workunit.client.0.smithi032.stdout:8 2024-09-13T12:22:20.071 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T12:22:20.071 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:22:20.071 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:22:20.484 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:22:21.485 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:22:21.485 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:22:21.485 INFO:tasks.workunit.client.0.smithi032.stdout:9 2024-09-13T12:22:21.486 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T12:22:21.486 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:22:21.486 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:22:22.030 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:22:23.031 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:22:23.031 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:22:23.032 INFO:tasks.workunit.client.0.smithi032.stdout:10 2024-09-13T12:22:23.032 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T12:22:23.032 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:22:23.032 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:22:23.443 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:22:24.444 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:22:24.445 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:22:24.445 INFO:tasks.workunit.client.0.smithi032.stdout:11 2024-09-13T12:22:24.446 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T12:22:24.446 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:22:24.446 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:22:24.866 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:22:25.868 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:22:25.868 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:22:25.868 INFO:tasks.workunit.client.0.smithi032.stdout:12 2024-09-13T12:22:25.869 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T12:22:25.869 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:22:25.869 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:22:26.288 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:22:27.290 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:22:27.290 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:22:27.290 INFO:tasks.workunit.client.0.smithi032.stdout:13 2024-09-13T12:22:27.291 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T12:22:27.291 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:22:27.291 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:22:28.284 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:22:29.285 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:22:29.286 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:22:29.286 INFO:tasks.workunit.client.0.smithi032.stdout:14 2024-09-13T12:22:29.286 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T12:22:29.286 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:22:29.287 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:22:30.056 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:22:31.058 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:22:31.058 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:22:31.058 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T12:22:31.058 INFO:tasks.workunit.client.0.smithi032.stdout:15 2024-09-13T12:22:31.059 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:22:31.059 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:22:31.822 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:22:32.824 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:22:32.824 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:22:32.824 INFO:tasks.workunit.client.0.smithi032.stdout:16 2024-09-13T12:22:32.825 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T12:22:32.825 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:22:32.825 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:22:33.801 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:22:34.803 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:22:34.803 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:22:34.803 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T12:22:34.803 INFO:tasks.workunit.client.0.smithi032.stdout:17 2024-09-13T12:22:34.804 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:22:34.804 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:22:35.486 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:22:36.488 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:22:36.488 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:22:36.488 INFO:tasks.workunit.client.0.smithi032.stdout:18 2024-09-13T12:22:36.489 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T12:22:36.489 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:22:36.489 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:22:37.250 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:22:38.252 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:22:38.252 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:22:38.252 INFO:tasks.workunit.client.0.smithi032.stdout:19 2024-09-13T12:22:38.253 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T12:22:38.253 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:22:38.253 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:22:38.712 INFO:tasks.workunit.client.0.smithi032.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1814077648,v1:127.0.0.1:6803/1814077648] [v2:127.0.0.1:6804/1814077648,v1:127.0.0.1:6805/1814077648] exists,up f0200d83-d430-4fd5-a4ea-2304279eaa4f 2024-09-13T12:22:38.712 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:22:38.713 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:22:38.713 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:22:38.713 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-13T12:22:38.717 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: osd_pid0=104325 2024-09-13T12:22:38.717 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:41: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 1 2024-09-13T12:22:38.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:22:38.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T12:22:38.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T12:22:38.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T12:22:38.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-13T12:22:38.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-13T12:22:38.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:22:38.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:22:38.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:22:38.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-13T12:22:38.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-13T12:22:38.719 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T12:22:38.719 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T12:22:38.719 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-13T12:22:38.719 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T12:22:38.719 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:22:38.719 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:22:38.719 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:22:38.719 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:22:38.719 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:22:38.719 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:22:38.719 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:22:38.719 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:22:38.719 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T12:22:38.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T12:22:38.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-13T12:22:38.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-13T12:22:38.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:22:38.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:22:38.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:22:38.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:22:38.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T12:22:38.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T12:22:38.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-13T12:22:38.720 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T12:22:38.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3bf12ce9-d5e2-4321-92bc-ee79874612e2 2024-09-13T12:22:38.721 INFO:tasks.workunit.client.0.smithi032.stdout:add osd1 3bf12ce9-d5e2-4321-92bc-ee79874612e2 2024-09-13T12:22:38.721 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 3bf12ce9-d5e2-4321-92bc-ee79874612e2' 2024-09-13T12:22:38.721 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T12:22:38.736 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCOLuRm/HHNKxAAkBnAsQTKCCe1v2wYFyewZQ== 2024-09-13T12:22:38.736 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCOLuRm/HHNKxAAkBnAsQTKCCe1v2wYFyewZQ=="}' 2024-09-13T12:22:38.736 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3bf12ce9-d5e2-4321-92bc-ee79874612e2 -i td/osd-bluefs-volume-ops/1/new.json 2024-09-13T12:22:39.269 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:22:39.280 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/1/new.json 2024-09-13T12:22:39.281 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCOLuRm/HHNKxAAkBnAsQTKCCe1v2wYFyewZQ== --osd-uuid 3bf12ce9-d5e2-4321-92bc-ee79874612e2 2024-09-13T12:22:39.309 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:22:39.307+0000 7fa229ca5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:22:39.311 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:22:39.309+0000 7fa229ca5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:22:39.312 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:22:39.310+0000 7fa229ca5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:22:39.312 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:22:39.311+0000 7fa229ca5640 -1 bdev(0x556b88b9dc00 td/osd-bluefs-volume-ops/1/block) open stat got: (1) Operation not permitted 2024-09-13T12:22:39.312 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:22:39.311+0000 7fa229ca5640 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_fsid unparsable uuid 2024-09-13T12:22:49.224 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/1/keyring 2024-09-13T12:22:49.225 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T12:22:49.225 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T12:22:49.226 INFO:tasks.workunit.client.0.smithi032.stdout:adding osd1 key to auth repository 2024-09-13T12:22:49.226 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T12:22:49.800 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.1 2024-09-13T12:22:49.801 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T12:22:49.801 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:22:49.801 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T12:22:49.801 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T12:22:49.801 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T12:22:49.826 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:22:49.825+0000 7ff416d89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:22:49.840 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:22:49.839+0000 7ff416d89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:22:49.841 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:22:49.840+0000 7ff416d89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:22:50.271 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T12:22:50.272 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:22:50.272 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T12:22:50.272 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:22:50.272 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:22:50.272 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:22:50.272 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:22:50.273 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:22:50.273 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:22:50.273 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:22:50.697 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:22:51.698 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:22:51.698 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:22:51.698 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:22:51.699 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:22:51.699 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:22:51.699 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:22:51.699 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:22:51.699+0000 7ff416d89640 -1 Falling back to public interface 2024-09-13T12:22:52.110 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:22:53.112 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:22:53.112 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:22:53.112 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:22:53.112 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:22:53.113 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:22:53.113 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:22:53.567 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:22:54.568 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:22:54.568 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:22:54.569 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:22:54.569 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:22:54.569 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:22:54.569 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:22:55.017 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:22:56.019 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:22:56.019 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:22:56.019 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:22:56.020 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:22:56.021 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:22:56.021 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:22:56.485 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:22:57.113 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:22:57.112+0000 7ff416d89640 -1 osd.1 0 log_to_monitors true 2024-09-13T12:22:57.487 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:22:57.487 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:22:57.487 INFO:tasks.workunit.client.0.smithi032.stdout:5 2024-09-13T12:22:57.487 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T12:22:57.488 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:22:57.488 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:22:58.022 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:22:59.023 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:22:59.023 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:22:59.023 INFO:tasks.workunit.client.0.smithi032.stdout:6 2024-09-13T12:22:59.024 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T12:22:59.024 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:22:59.024 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:22:59.481 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:00.482 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:00.482 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:00.483 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T12:23:00.483 INFO:tasks.workunit.client.0.smithi032.stdout:7 2024-09-13T12:23:00.483 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:00.483 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:23:01.025 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:02.026 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:02.026 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:02.026 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T12:23:02.027 INFO:tasks.workunit.client.0.smithi032.stdout:8 2024-09-13T12:23:02.027 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:02.027 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:23:02.526 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:03.528 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:03.528 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:03.528 INFO:tasks.workunit.client.0.smithi032.stdout:9 2024-09-13T12:23:03.529 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T12:23:03.529 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:03.529 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:23:04.028 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:05.029 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:05.029 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:05.029 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T12:23:05.030 INFO:tasks.workunit.client.0.smithi032.stdout:10 2024-09-13T12:23:05.030 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:05.030 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:23:05.472 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:06.473 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:06.473 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:06.474 INFO:tasks.workunit.client.0.smithi032.stdout:11 2024-09-13T12:23:06.474 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T12:23:06.474 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:06.474 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:23:06.987 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:07.988 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:07.988 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:07.989 INFO:tasks.workunit.client.0.smithi032.stdout:12 2024-09-13T12:23:07.989 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T12:23:07.989 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:07.989 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:23:08.404 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:09.406 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:09.406 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:09.406 INFO:tasks.workunit.client.0.smithi032.stdout:13 2024-09-13T12:23:09.406 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T12:23:09.407 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:09.407 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:23:10.013 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:11.015 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:11.015 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:11.015 INFO:tasks.workunit.client.0.smithi032.stdout:14 2024-09-13T12:23:11.016 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T12:23:11.016 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:11.016 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:23:11.795 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:12.797 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:12.797 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:12.797 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T12:23:12.797 INFO:tasks.workunit.client.0.smithi032.stdout:15 2024-09-13T12:23:12.798 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:12.798 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:23:13.884 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:14.885 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:14.886 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:14.886 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T12:23:14.886 INFO:tasks.workunit.client.0.smithi032.stdout:16 2024-09-13T12:23:14.886 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:14.886 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:23:15.895 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:16.896 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:16.897 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:16.897 INFO:tasks.workunit.client.0.smithi032.stdout:17 2024-09-13T12:23:16.897 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T12:23:16.897 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:16.898 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:23:17.991 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:18.993 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:18.993 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:18.993 INFO:tasks.workunit.client.0.smithi032.stdout:18 2024-09-13T12:23:18.994 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T12:23:18.994 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:18.994 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:23:19.527 INFO:tasks.workunit.client.0.smithi032.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1651970145,v1:127.0.0.1:6811/1651970145] [v2:127.0.0.1:6812/1651970145,v1:127.0.0.1:6813/1651970145] exists,up 3bf12ce9-d5e2-4321-92bc-ee79874612e2 2024-09-13T12:23:19.527 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:23:19.528 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:23:19.528 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:23:19.528 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-13T12:23:19.532 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: osd_pid1=106879 2024-09-13T12:23:19.532 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:43: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 2 2024-09-13T12:23:19.532 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:23:19.532 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T12:23:19.532 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T12:23:19.532 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T12:23:19.532 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-13T12:23:19.532 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-13T12:23:19.532 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:23:19.532 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:23:19.533 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:23:19.533 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-13T12:23:19.533 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-13T12:23:19.533 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T12:23:19.533 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T12:23:19.533 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-13T12:23:19.533 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T12:23:19.533 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:23:19.533 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:23:19.533 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:23:19.533 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:23:19.533 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:23:19.533 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:23:19.534 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:23:19.534 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:23:19.534 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T12:23:19.534 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T12:23:19.534 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-13T12:23:19.534 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-13T12:23:19.534 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:23:19.534 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:23:19.534 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:23:19.534 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:23:19.534 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T12:23:19.534 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T12:23:19.534 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-13T12:23:19.534 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T12:23:19.535 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=78431f38-564b-40c5-9ca6-e92448355c2a 2024-09-13T12:23:19.535 INFO:tasks.workunit.client.0.smithi032.stdout:add osd2 78431f38-564b-40c5-9ca6-e92448355c2a 2024-09-13T12:23:19.535 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 78431f38-564b-40c5-9ca6-e92448355c2a' 2024-09-13T12:23:19.536 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T12:23:19.551 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC3LuRml5fFIBAA2FuDmKQ9TsbsOQSEB7Y/rA== 2024-09-13T12:23:19.551 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC3LuRml5fFIBAA2FuDmKQ9TsbsOQSEB7Y/rA=="}' 2024-09-13T12:23:19.551 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 78431f38-564b-40c5-9ca6-e92448355c2a -i td/osd-bluefs-volume-ops/2/new.json 2024-09-13T12:23:20.120 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:23:20.131 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/2/new.json 2024-09-13T12:23:20.132 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC3LuRml5fFIBAA2FuDmKQ9TsbsOQSEB7Y/rA== --osd-uuid 78431f38-564b-40c5-9ca6-e92448355c2a 2024-09-13T12:23:20.161 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:23:20.159+0000 7f0d0ce14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:23:20.178 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:23:20.176+0000 7f0d0ce14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:23:20.179 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:23:20.177+0000 7f0d0ce14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:23:20.179 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:23:20.178+0000 7f0d0ce14640 -1 bdev(0x564d237d3c00 td/osd-bluefs-volume-ops/2/block) open stat got: (1) Operation not permitted 2024-09-13T12:23:20.179 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:23:20.178+0000 7f0d0ce14640 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_fsid unparsable uuid 2024-09-13T12:23:29.174 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/2/keyring 2024-09-13T12:23:29.175 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T12:23:29.175 INFO:tasks.workunit.client.0.smithi032.stdout:adding osd2 key to auth repository 2024-09-13T12:23:29.176 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T12:23:29.176 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T12:23:29.737 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.2 2024-09-13T12:23:29.738 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T12:23:29.738 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:23:29.738 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T12:23:29.738 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T12:23:29.739 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T12:23:29.764 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:23:29.764+0000 7f16c53a4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:23:29.774 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:23:29.773+0000 7f16c53a4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:23:29.775 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:23:29.774+0000 7f16c53a4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:23:30.220 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T12:23:30.221 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:23:30.221 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T12:23:30.221 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:23:30.221 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:23:30.221 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:30.221 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:23:30.222 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:23:30.222 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:30.222 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:23:30.689 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:31.634 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:23:31.633+0000 7f16c53a4640 -1 Falling back to public interface 2024-09-13T12:23:31.690 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:31.690 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:31.690 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:23:31.691 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:23:31.691 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:31.691 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:23:32.149 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:33.150 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:33.150 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:33.150 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:23:33.151 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:23:33.151 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:33.151 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:23:33.583 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:34.585 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:34.585 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:34.585 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:23:34.585 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:23:34.586 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:34.586 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:23:35.018 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:36.019 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:36.019 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:36.020 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:23:36.020 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:23:36.020 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:36.020 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:23:36.368 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:23:36.367+0000 7f16c53a4640 -1 osd.2 0 log_to_monitors true 2024-09-13T12:23:36.488 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:37.489 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:37.490 INFO:tasks.workunit.client.0.smithi032.stdout:5 2024-09-13T12:23:37.490 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:37.490 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T12:23:37.490 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:37.490 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:23:37.902 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:38.904 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:38.904 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:38.904 INFO:tasks.workunit.client.0.smithi032.stdout:6 2024-09-13T12:23:38.905 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T12:23:38.905 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:38.905 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:23:39.398 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:40.400 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:40.400 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:40.400 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T12:23:40.400 INFO:tasks.workunit.client.0.smithi032.stdout:7 2024-09-13T12:23:40.401 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:40.401 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:23:40.829 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:41.830 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:41.830 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:41.830 INFO:tasks.workunit.client.0.smithi032.stdout:8 2024-09-13T12:23:41.831 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T12:23:41.831 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:41.831 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:23:42.353 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:43.354 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:43.355 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:43.355 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T12:23:43.355 INFO:tasks.workunit.client.0.smithi032.stdout:9 2024-09-13T12:23:43.355 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:43.356 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:23:43.772 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:44.774 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:44.774 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:44.774 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T12:23:44.775 INFO:tasks.workunit.client.0.smithi032.stdout:10 2024-09-13T12:23:44.775 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:44.775 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:23:45.285 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:46.287 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:46.287 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:46.287 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T12:23:46.287 INFO:tasks.workunit.client.0.smithi032.stdout:11 2024-09-13T12:23:46.288 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:46.288 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:23:46.701 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:47.702 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:47.702 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:47.702 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T12:23:47.702 INFO:tasks.workunit.client.0.smithi032.stdout:12 2024-09-13T12:23:47.703 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:47.703 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:23:48.115 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:49.116 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:49.116 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:49.116 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T12:23:49.116 INFO:tasks.workunit.client.0.smithi032.stdout:13 2024-09-13T12:23:49.117 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:49.117 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:23:49.865 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:50.867 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:50.867 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:50.867 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T12:23:50.867 INFO:tasks.workunit.client.0.smithi032.stdout:14 2024-09-13T12:23:50.868 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:50.868 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:23:51.602 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:52.603 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:52.603 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:52.603 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T12:23:52.603 INFO:tasks.workunit.client.0.smithi032.stdout:15 2024-09-13T12:23:52.604 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:52.604 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:23:53.724 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:54.726 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:54.726 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:54.726 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T12:23:54.726 INFO:tasks.workunit.client.0.smithi032.stdout:16 2024-09-13T12:23:54.726 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:54.727 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:23:55.668 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:56.670 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:56.670 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:56.670 INFO:tasks.workunit.client.0.smithi032.stdout:17 2024-09-13T12:23:56.671 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T12:23:56.671 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:56.671 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:23:58.013 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:23:59.014 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:23:59.014 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:23:59.014 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T12:23:59.015 INFO:tasks.workunit.client.0.smithi032.stdout:18 2024-09-13T12:23:59.015 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:23:59.015 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:23:59.476 INFO:tasks.workunit.client.0.smithi032.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1995340593,v1:127.0.0.1:6819/1995340593] [v2:127.0.0.1:6820/1995340593,v1:127.0.0.1:6821/1995340593] exists,up 78431f38-564b-40c5-9ca6-e92448355c2a 2024-09-13T12:23:59.476 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:23:59.477 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:23:59.477 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:23:59.477 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-13T12:23:59.480 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: osd_pid2=109408 2024-09-13T12:23:59.480 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:45: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 3 2024-09-13T12:23:59.480 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:23:59.480 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T12:23:59.480 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-13T12:23:59.480 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T12:23:59.481 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-13T12:23:59.481 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-13T12:23:59.481 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:23:59.481 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:23:59.481 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:23:59.481 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-13T12:23:59.481 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-13T12:23:59.481 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T12:23:59.481 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T12:23:59.481 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-13T12:23:59.481 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T12:23:59.481 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:23:59.481 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:23:59.482 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:23:59.482 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:23:59.482 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:23:59.482 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:23:59.482 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:23:59.482 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:23:59.482 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T12:23:59.482 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T12:23:59.482 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-13T12:23:59.482 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-13T12:23:59.482 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:23:59.482 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:23:59.482 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:23:59.483 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:23:59.483 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T12:23:59.483 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T12:23:59.483 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-13T12:23:59.483 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T12:23:59.483 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fc41ef3f-07fe-4ce3-aecb-2e8f6a9edb19 2024-09-13T12:23:59.484 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 fc41ef3f-07fe-4ce3-aecb-2e8f6a9edb19' 2024-09-13T12:23:59.484 INFO:tasks.workunit.client.0.smithi032.stdout:add osd3 fc41ef3f-07fe-4ce3-aecb-2e8f6a9edb19 2024-09-13T12:23:59.484 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T12:23:59.499 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDfLuRmypW5HRAAm5pfh+JAxPPunRDedAGF2A== 2024-09-13T12:23:59.499 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDfLuRmypW5HRAAm5pfh+JAxPPunRDedAGF2A=="}' 2024-09-13T12:23:59.500 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fc41ef3f-07fe-4ce3-aecb-2e8f6a9edb19 -i td/osd-bluefs-volume-ops/3/new.json 2024-09-13T12:23:59.965 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:23:59.977 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/3/new.json 2024-09-13T12:23:59.978 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDfLuRmypW5HRAAm5pfh+JAxPPunRDedAGF2A== --osd-uuid fc41ef3f-07fe-4ce3-aecb-2e8f6a9edb19 2024-09-13T12:24:00.006 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:24:00.005+0000 7f0ff941c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:24:00.019 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:24:00.018+0000 7f0ff941c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:24:00.020 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:24:00.019+0000 7f0ff941c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:24:00.020 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:24:00.020+0000 7f0ff941c640 -1 bdev(0x562dceaa3c00 td/osd-bluefs-volume-ops/3/block) open stat got: (1) Operation not permitted 2024-09-13T12:24:00.020 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:24:00.020+0000 7f0ff941c640 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_fsid unparsable uuid 2024-09-13T12:24:09.955 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/3/keyring 2024-09-13T12:24:09.955 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T12:24:09.956 INFO:tasks.workunit.client.0.smithi032.stdout:adding osd3 key to auth repository 2024-09-13T12:24:09.956 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-13T12:24:09.957 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T12:24:10.519 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.3 2024-09-13T12:24:10.520 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-13T12:24:10.520 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:24:10.520 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T12:24:10.520 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T12:24:10.520 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T12:24:10.546 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:24:10.544+0000 7fae15687640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:24:10.548 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:24:10.546+0000 7fae15687640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:24:10.549 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:24:10.548+0000 7fae15687640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:24:10.970 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-13T12:24:10.970 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:24:10.970 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T12:24:10.970 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:24:10.970 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:24:10.970 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:24:10.971 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:24:10.971 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:24:10.971 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:24:10.971 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:24:11.395 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:24:11.894 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:24:11.892+0000 7fae15687640 -1 Falling back to public interface 2024-09-13T12:24:12.397 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:24:12.397 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:24:12.397 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:24:12.397 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:24:12.398 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:24:12.398 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:24:12.839 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:24:13.840 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:24:13.840 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:24:13.840 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:24:13.841 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:24:13.841 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:24:13.841 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:24:14.290 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:24:15.291 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:24:15.292 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:24:15.292 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:24:15.292 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:24:15.292 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:24:15.293 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:24:15.725 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:24:16.726 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:24:16.726 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:24:16.726 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:24:16.726 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:24:16.727 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:24:16.727 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:24:17.015 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:24:17.014+0000 7fae15687640 -1 osd.3 0 log_to_monitors true 2024-09-13T12:24:17.185 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:24:18.186 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:24:18.187 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:24:18.187 INFO:tasks.workunit.client.0.smithi032.stdout:5 2024-09-13T12:24:18.187 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T12:24:18.187 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:24:18.188 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:24:18.609 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:24:19.610 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:24:19.610 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:24:19.610 INFO:tasks.workunit.client.0.smithi032.stdout:6 2024-09-13T12:24:19.611 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T12:24:19.611 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:24:19.611 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:24:20.019 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:24:21.021 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:24:21.021 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:24:21.021 INFO:tasks.workunit.client.0.smithi032.stdout:7 2024-09-13T12:24:21.022 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T12:24:21.022 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:24:21.022 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:24:21.446 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:24:22.448 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:24:22.448 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:24:22.448 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T12:24:22.448 INFO:tasks.workunit.client.0.smithi032.stdout:8 2024-09-13T12:24:22.449 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:24:22.449 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:24:22.967 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:24:23.968 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:24:23.968 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:24:23.968 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T12:24:23.969 INFO:tasks.workunit.client.0.smithi032.stdout:9 2024-09-13T12:24:23.969 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:24:23.969 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:24:24.402 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:24:25.404 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:24:25.404 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:24:25.405 INFO:tasks.workunit.client.0.smithi032.stdout:10 2024-09-13T12:24:25.405 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T12:24:25.406 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:24:25.406 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:24:25.861 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:24:26.863 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:24:26.863 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:24:26.863 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T12:24:26.863 INFO:tasks.workunit.client.0.smithi032.stdout:11 2024-09-13T12:24:26.864 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:24:26.864 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:24:27.291 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:24:28.293 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:24:28.293 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:24:28.293 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T12:24:28.293 INFO:tasks.workunit.client.0.smithi032.stdout:12 2024-09-13T12:24:28.294 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:24:28.294 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:24:28.710 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:24:29.711 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:24:29.711 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:24:29.711 INFO:tasks.workunit.client.0.smithi032.stdout:13 2024-09-13T12:24:29.712 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T12:24:29.712 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:24:29.712 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:24:30.140 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:24:31.141 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:24:31.141 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:24:31.142 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T12:24:31.142 INFO:tasks.workunit.client.0.smithi032.stdout:14 2024-09-13T12:24:31.142 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:24:31.142 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:24:31.848 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:24:32.849 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:24:32.849 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:24:32.849 INFO:tasks.workunit.client.0.smithi032.stdout:15 2024-09-13T12:24:32.850 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T12:24:32.850 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:24:32.850 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:24:33.470 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:24:34.472 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:24:34.472 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:24:34.472 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T12:24:34.472 INFO:tasks.workunit.client.0.smithi032.stdout:16 2024-09-13T12:24:34.473 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:24:34.473 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:24:35.194 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:24:36.196 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:24:36.196 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:24:36.196 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T12:24:36.196 INFO:tasks.workunit.client.0.smithi032.stdout:17 2024-09-13T12:24:36.197 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:24:36.197 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:24:36.621 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:24:37.623 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:24:37.623 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:24:37.623 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T12:24:37.623 INFO:tasks.workunit.client.0.smithi032.stdout:18 2024-09-13T12:24:37.624 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:24:37.624 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:24:38.276 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:24:39.277 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:24:39.277 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:24:39.277 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T12:24:39.277 INFO:tasks.workunit.client.0.smithi032.stdout:19 2024-09-13T12:24:39.278 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:24:39.278 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:24:39.843 INFO:tasks.workunit.client.0.smithi032.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3005265510,v1:127.0.0.1:6827/3005265510] [v2:127.0.0.1:6828/3005265510,v1:127.0.0.1:6829/3005265510] exists,up fc41ef3f-07fe-4ce3-aecb-2e8f6a9edb19 2024-09-13T12:24:39.844 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:24:39.844 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:24:39.844 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:24:39.844 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-13T12:24:39.844 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: osd_pid3=111937 2024-09-13T12:24:39.845 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:48: TEST_bluestore: sleep 5 2024-09-13T12:24:44.846 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:50: TEST_bluestore: create_pool foo 16 2024-09-13T12:24:44.846 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 16 2024-09-13T12:24:45.606 INFO:tasks.workunit.client.0.smithi032.stderr:pool 'foo' already exists 2024-09-13T12:24:45.618 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T12:24:46.619 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:53: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-13T12:24:46.719 INFO:tasks.workunit.client.0.smithi032.stdout:hints = 1 2024-09-13T12:24:46.719 INFO:tasks.workunit.client.0.smithi032.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-13T12:24:46.719 INFO:tasks.workunit.client.0.smithi032.stdout:Object prefix: benchmark_data_smithi032_113303 2024-09-13T12:24:46.719 INFO:tasks.workunit.client.0.smithi032.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T12:24:46.719 INFO:tasks.workunit.client.0.smithi032.stdout: 0 0 0 0 0 0 - 0 2024-09-13T12:24:47.719 INFO:tasks.workunit.client.0.smithi032.stdout: 1 16 64 48 0.187485 0.1875 0.390201 0.261371 2024-09-13T12:24:48.719 INFO:tasks.workunit.client.0.smithi032.stdout: 2 16 120 104 0.203108 0.21875 0.299838 0.284195 2024-09-13T12:24:49.720 INFO:tasks.workunit.client.0.smithi032.stdout: 3 16 184 168 0.218727 0.25 0.266751 0.270035 2024-09-13T12:24:50.719 INFO:tasks.workunit.client.0.smithi032.stdout: 4 16 248 232 0.226539 0.25 0.228518 0.265882 2024-09-13T12:24:51.720 INFO:tasks.workunit.client.0.smithi032.stdout: 5 16 311 295 0.230444 0.246094 0.245222 0.266058 2024-09-13T12:24:52.720 INFO:tasks.workunit.client.0.smithi032.stdout: 6 16 375 359 0.233697 0.25 0.257794 0.262999 2024-09-13T12:24:53.720 INFO:tasks.workunit.client.0.smithi032.stdout: 7 16 425 409 0.228209 0.195312 0.250167 0.264457 2024-09-13T12:24:54.720 INFO:tasks.workunit.client.0.smithi032.stdout: 8 16 473 457 0.223116 0.1875 0.240007 0.276254 2024-09-13T12:24:55.720 INFO:tasks.workunit.client.0.smithi032.stdout: 9 16 538 522 0.226534 0.253906 0.173425 0.272295 2024-09-13T12:24:56.720 INFO:tasks.workunit.client.0.smithi032.stdout: 10 16 603 587 0.229268 0.253906 0.218487 0.268999 2024-09-13T12:24:57.721 INFO:tasks.workunit.client.0.smithi032.stdout: 11 16 665 649 0.230438 0.242188 0.319266 0.268119 2024-09-13T12:24:58.721 INFO:tasks.workunit.client.0.smithi032.stdout: 12 16 729 713 0.232066 0.25 0.261808 0.26751 2024-09-13T12:24:59.721 INFO:tasks.workunit.client.0.smithi032.stdout: 13 16 782 766 0.230138 0.207031 0.243557 0.265927 2024-09-13T12:25:00.721 INFO:tasks.workunit.client.0.smithi032.stdout: 14 16 846 830 0.231554 0.25 0.251588 0.267365 2024-09-13T12:25:01.721 INFO:tasks.workunit.client.0.smithi032.stdout: 15 16 910 894 0.232781 0.25 0.242515 0.266403 2024-09-13T12:25:02.721 INFO:tasks.workunit.client.0.smithi032.stdout: 16 16 969 953 0.232635 0.230469 0.303585 0.266511 2024-09-13T12:25:03.722 INFO:tasks.workunit.client.0.smithi032.stdout: 17 16 1033 1017 0.233655 0.25 0.250721 0.265622 2024-09-13T12:25:04.721 INFO:tasks.workunit.client.0.smithi032.stdout: 18 16 1097 1081 0.234561 0.25 0.251208 0.264325 2024-09-13T12:25:05.722 INFO:tasks.workunit.client.0.smithi032.stdout: 19 16 1150 1134 0.233111 0.207031 0.386922 0.266227 2024-09-13T12:25:06.722 INFO:tasks.workunit.client.0.smithi032.stdout:2024-09-13T12:25:06.722146+0000 min lat: 0.173425 max lat: 0.617277 avg lat: 0.26548 2024-09-13T12:25:06.722 INFO:tasks.workunit.client.0.smithi032.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T12:25:06.722 INFO:tasks.workunit.client.0.smithi032.stdout: 20 16 1214 1198 0.233953 0.25 0.237565 0.26548 2024-09-13T12:25:07.722 INFO:tasks.workunit.client.0.smithi032.stdout: 21 16 1278 1262 0.234716 0.25 0.27732 0.265701 2024-09-13T12:25:08.722 INFO:tasks.workunit.client.0.smithi032.stdout: 22 16 1342 1326 0.235409 0.25 0.276148 0.264849 2024-09-13T12:25:09.722 INFO:tasks.workunit.client.0.smithi032.stdout: 23 16 1394 1378 0.234004 0.203125 0.267172 0.26485 2024-09-13T12:25:10.722 INFO:tasks.workunit.client.0.smithi032.stdout: 24 16 1458 1442 0.234669 0.25 0.258813 0.264272 2024-09-13T12:25:11.722 INFO:tasks.workunit.client.0.smithi032.stdout: 25 16 1522 1506 0.235282 0.25 0.294004 0.263689 2024-09-13T12:25:12.723 INFO:tasks.workunit.client.0.smithi032.stdout: 26 16 1586 1570 0.235847 0.25 0.256073 0.263962 2024-09-13T12:25:13.723 INFO:tasks.workunit.client.0.smithi032.stdout: 27 16 1646 1630 0.235791 0.234375 0.260895 0.263731 2024-09-13T12:25:14.723 INFO:tasks.workunit.client.0.smithi032.stdout: 28 16 1710 1694 0.236298 0.25 0.277274 0.26346 2024-09-13T12:25:15.723 INFO:tasks.workunit.client.0.smithi032.stdout: 29 16 1774 1758 0.236769 0.25 0.261326 0.263507 2024-09-13T12:25:16.723 INFO:tasks.workunit.client.0.smithi032.stdout: 30 16 1826 1810 0.235647 0.203125 0.250577 0.263269 2024-09-13T12:25:17.723 INFO:tasks.workunit.client.0.smithi032.stdout:Total time run: 30.1276 2024-09-13T12:25:17.723 INFO:tasks.workunit.client.0.smithi032.stdout:Total writes made: 1826 2024-09-13T12:25:17.724 INFO:tasks.workunit.client.0.smithi032.stdout:Write size: 4096 2024-09-13T12:25:17.724 INFO:tasks.workunit.client.0.smithi032.stdout:Object size: 4096 2024-09-13T12:25:17.724 INFO:tasks.workunit.client.0.smithi032.stdout:Bandwidth (MB/sec): 0.236754 2024-09-13T12:25:17.724 INFO:tasks.workunit.client.0.smithi032.stdout:Stddev Bandwidth: 0.0223182 2024-09-13T12:25:17.724 INFO:tasks.workunit.client.0.smithi032.stdout:Max bandwidth (MB/sec): 0.253906 2024-09-13T12:25:17.724 INFO:tasks.workunit.client.0.smithi032.stdout:Min bandwidth (MB/sec): 0.1875 2024-09-13T12:25:17.724 INFO:tasks.workunit.client.0.smithi032.stdout:Average IOPS: 60 2024-09-13T12:25:17.724 INFO:tasks.workunit.client.0.smithi032.stdout:Stddev IOPS: 5.71346 2024-09-13T12:25:17.724 INFO:tasks.workunit.client.0.smithi032.stdout:Max IOPS: 65 2024-09-13T12:25:17.724 INFO:tasks.workunit.client.0.smithi032.stdout:Min IOPS: 48 2024-09-13T12:25:17.724 INFO:tasks.workunit.client.0.smithi032.stdout:Average Latency(s): 0.263266 2024-09-13T12:25:17.724 INFO:tasks.workunit.client.0.smithi032.stdout:Stddev Latency(s): 0.0440446 2024-09-13T12:25:17.724 INFO:tasks.workunit.client.0.smithi032.stdout:Max latency(s): 0.617277 2024-09-13T12:25:17.725 INFO:tasks.workunit.client.0.smithi032.stdout:Min latency(s): 0.173425 2024-09-13T12:25:17.843 INFO:tasks.workunit.client.0.smithi032.stdout:after bench 2024-09-13T12:25:17.844 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:55: TEST_bluestore: echo 'after bench' 2024-09-13T12:25:17.844 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 104325 2024-09-13T12:25:17.844 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: sleep 1 2024-09-13T12:25:18.845 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 104325 2024-09-13T12:25:18.845 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 58: kill: (104325) - No such process 2024-09-13T12:25:18.845 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:59: TEST_bluestore: ceph osd down 0 2024-09-13T12:25:19.380 INFO:tasks.workunit.client.0.smithi032.stderr:osd.0 is already down. 2024-09-13T12:25:19.391 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 106879 2024-09-13T12:25:19.391 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: sleep 1 2024-09-13T12:25:20.393 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 106879 2024-09-13T12:25:20.393 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 60: kill: (106879) - No such process 2024-09-13T12:25:20.393 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:61: TEST_bluestore: ceph osd down 1 2024-09-13T12:25:20.901 INFO:tasks.workunit.client.0.smithi032.stderr:osd.1 is already down. 2024-09-13T12:25:20.913 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 109408 2024-09-13T12:25:20.913 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: sleep 1 2024-09-13T12:25:21.914 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 109408 2024-09-13T12:25:21.915 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 62: kill: (109408) - No such process 2024-09-13T12:25:21.915 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:63: TEST_bluestore: ceph osd down 2 2024-09-13T12:25:22.361 INFO:tasks.workunit.client.0.smithi032.stderr:osd.2 is already down. 2024-09-13T12:25:22.373 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 111937 2024-09-13T12:25:22.373 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: sleep 1 2024-09-13T12:25:23.375 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 111937 2024-09-13T12:25:23.375 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 64: kill: (111937) - No such process 2024-09-13T12:25:23.375 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:65: TEST_bluestore: ceph osd down 3 2024-09-13T12:25:23.892 INFO:tasks.workunit.client.0.smithi032.stderr:osd.3 is already down. 2024-09-13T12:25:23.904 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:68: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-13T12:25:26.468 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:25:26.479 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:69: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-13T12:25:28.553 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:25:28.563 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:70: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-13T12:25:30.885 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:25:30.897 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:71: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-13T12:25:33.208 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:25:33.219 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:73: TEST_bluestore: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2024-09-13T12:25:33.229 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:74: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2024-09-13T12:25:33.254 INFO:tasks.workunit.client.0.smithi032.stdout:inferring bluefs devices from bluestore path 2024-09-13T12:25:34.361 INFO:tasks.workunit.client.0.smithi032.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-09-13T12:25:34.361 INFO:tasks.workunit.client.0.smithi032.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-13T12:25:34.361 INFO:tasks.workunit.client.0.smithi032.stdout:2 : device size 0x100000000 : using 0x80583000(2.0 GiB) 2024-09-13T12:25:34.361 INFO:tasks.workunit.client.0.smithi032.stdout:Expanding DB/WAL... 2024-09-13T12:25:34.361 INFO:tasks.workunit.client.0.smithi032.stdout:0 : expanding to 0x536870912 2024-09-13T12:25:34.370 INFO:tasks.workunit.client.0.smithi032.stdout:0 : size label updated to 536870912 2024-09-13T12:25:34.370 INFO:tasks.workunit.client.0.smithi032.stdout:1 : expanding to 0x1073741824 2024-09-13T12:25:34.379 INFO:tasks.workunit.client.0.smithi032.stdout:1 : size label updated to 1073741824 2024-09-13T12:25:34.379 INFO:tasks.workunit.client.0.smithi032.stdout:2 : expanding from 0x80000000 to 0x100000000 2024-09-13T12:25:34.537 INFO:tasks.workunit.client.0.smithi032.stdout:2 : size label updated to 4294967296 2024-09-13T12:25:39.564 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:75: TEST_bluestore: truncate td/osd-bluefs-volume-ops/1/block -s 4311744512 2024-09-13T12:25:39.564 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:76: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 bluefs-bdev-expand 2024-09-13T12:25:39.589 INFO:tasks.workunit.client.0.smithi032.stdout:inferring bluefs devices from bluestore path 2024-09-13T12:25:40.703 INFO:tasks.workunit.client.0.smithi032.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-09-13T12:25:40.703 INFO:tasks.workunit.client.0.smithi032.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-13T12:25:40.703 INFO:tasks.workunit.client.0.smithi032.stdout:2 : device size 0x101000000 : using 0x815a1000(2.0 GiB) 2024-09-13T12:25:40.703 INFO:tasks.workunit.client.0.smithi032.stdout:Expanding DB/WAL... 2024-09-13T12:25:40.703 INFO:tasks.workunit.client.0.smithi032.stdout:0 : expanding to 0x536870912 2024-09-13T12:25:40.709 INFO:tasks.workunit.client.0.smithi032.stdout:0 : size label updated to 536870912 2024-09-13T12:25:40.709 INFO:tasks.workunit.client.0.smithi032.stdout:1 : expanding to 0x1073741824 2024-09-13T12:25:40.717 INFO:tasks.workunit.client.0.smithi032.stdout:1 : size label updated to 1073741824 2024-09-13T12:25:40.717 INFO:tasks.workunit.client.0.smithi032.stdout:2 : expanding from 0x80000000 to 0x101000000 2024-09-13T12:25:40.842 INFO:tasks.workunit.client.0.smithi032.stdout:2 : size label updated to 4311744512 2024-09-13T12:25:45.954 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:77: TEST_bluestore: truncate td/osd-bluefs-volume-ops/2/block -s 4295099392 2024-09-13T12:25:45.955 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:78: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 bluefs-bdev-expand 2024-09-13T12:25:45.979 INFO:tasks.workunit.client.0.smithi032.stdout:inferring bluefs devices from bluestore path 2024-09-13T12:25:47.150 INFO:tasks.workunit.client.0.smithi032.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-09-13T12:25:47.151 INFO:tasks.workunit.client.0.smithi032.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-13T12:25:47.151 INFO:tasks.workunit.client.0.smithi032.stdout:2 : device size 0x100020000 : using 0x805c1000(2.0 GiB) 2024-09-13T12:25:47.151 INFO:tasks.workunit.client.0.smithi032.stdout:Expanding DB/WAL... 2024-09-13T12:25:47.151 INFO:tasks.workunit.client.0.smithi032.stdout:0 : expanding to 0x536870912 2024-09-13T12:25:47.158 INFO:tasks.workunit.client.0.smithi032.stdout:0 : size label updated to 536870912 2024-09-13T12:25:47.158 INFO:tasks.workunit.client.0.smithi032.stdout:1 : expanding to 0x1073741824 2024-09-13T12:25:47.166 INFO:tasks.workunit.client.0.smithi032.stdout:1 : size label updated to 1073741824 2024-09-13T12:25:47.167 INFO:tasks.workunit.client.0.smithi032.stdout:2 : expanding from 0x80000000 to 0x100020000 2024-09-13T12:25:47.292 INFO:tasks.workunit.client.0.smithi032.stdout:2 : size label updated to 4295098368 2024-09-13T12:25:52.189 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:79: TEST_bluestore: truncate td/osd-bluefs-volume-ops/3/block -s 4293918720 2024-09-13T12:25:52.190 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:80: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 bluefs-bdev-expand 2024-09-13T12:25:52.214 INFO:tasks.workunit.client.0.smithi032.stdout:inferring bluefs devices from bluestore path 2024-09-13T12:25:53.148 INFO:tasks.workunit.client.0.smithi032.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-09-13T12:25:53.148 INFO:tasks.workunit.client.0.smithi032.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-13T12:25:53.148 INFO:tasks.workunit.client.0.smithi032.stdout:2 : device size 0xfff00000 : using 0x804b2000(2.0 GiB) 2024-09-13T12:25:53.148 INFO:tasks.workunit.client.0.smithi032.stdout:Expanding DB/WAL... 2024-09-13T12:25:53.148 INFO:tasks.workunit.client.0.smithi032.stdout:0 : expanding to 0x536870912 2024-09-13T12:25:53.156 INFO:tasks.workunit.client.0.smithi032.stdout:0 : size label updated to 536870912 2024-09-13T12:25:53.156 INFO:tasks.workunit.client.0.smithi032.stdout:1 : expanding to 0x1073741824 2024-09-13T12:25:53.163 INFO:tasks.workunit.client.0.smithi032.stdout:1 : size label updated to 1073741824 2024-09-13T12:25:53.164 INFO:tasks.workunit.client.0.smithi032.stdout:2 : expanding from 0x80000000 to 0xfff00000 2024-09-13T12:25:53.322 INFO:tasks.workunit.client.0.smithi032.stdout:2 : size label updated to 4293918720 2024-09-13T12:25:58.468 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:83: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-13T12:26:00.533 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:26:00.543 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:84: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-13T12:26:02.862 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:26:02.871 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:85: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-13T12:26:05.181 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:26:05.190 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:86: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-13T12:26:07.504 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:26:07.515 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:88: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2024-09-13T12:26:07.539 INFO:tasks.workunit.client.0.smithi032.stdout:inferring bluefs devices from bluestore path 2024-09-13T12:26:09.566 INFO:tasks.workunit.client.0.smithi032.stdout:0 : device size 0x1ffff000 : using 0x1600000(22 MiB) 2024-09-13T12:26:09.566 INFO:tasks.workunit.client.0.smithi032.stdout:1 : device size 0x3fffe000 : using 0x1600000(22 MiB) 2024-09-13T12:26:09.566 INFO:tasks.workunit.client.0.smithi032.stdout:2 : device size 0x100000000 : using 0x583000(5.5 MiB) 2024-09-13T12:26:09.568 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:90: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/block.db --command bluefs-bdev-migrate 2024-09-13T12:26:09.591 INFO:tasks.workunit.client.0.smithi032.stdout:inferring bluefs devices from bluestore path 2024-09-13T12:26:12.157 INFO:tasks.workunit.client.0.smithi032.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-09-13T12:26:12.162 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:95: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-13T12:26:13.978 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:26:13.988 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:98: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-09-13T12:26:14.013 INFO:tasks.workunit.client.0.smithi032.stdout:inferring bluefs devices from bluestore path 2024-09-13T12:26:16.828 INFO:tasks.workunit.client.0.smithi032.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-09-13T12:26:16.832 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:103: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-13T12:26:18.656 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:26:18.665 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:106: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block.wal --devs-source td/osd-bluefs-volume-ops/2/block.db --dev-target td/osd-bluefs-volume-ops/2/block --command bluefs-bdev-migrate 2024-09-13T12:26:18.689 INFO:tasks.workunit.client.0.smithi032.stdout:inferring bluefs devices from bluestore path 2024-09-13T12:26:21.502 INFO:tasks.workunit.client.0.smithi032.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-09-13T12:26:21.502 INFO:tasks.workunit.client.0.smithi032.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-09-13T12:26:21.507 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:112: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-13T12:26:22.826 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:26:22.834 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:115: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block.wal --command bluefs-bdev-migrate 2024-09-13T12:26:22.856 INFO:tasks.workunit.client.0.smithi032.stdout:inferring bluefs devices from bluestore path 2024-09-13T12:26:23.658 INFO:tasks.workunit.client.0.smithi032.stderr:Migrate to WAL device isn't supported. 2024-09-13T12:26:23.660 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:121: TEST_bluestore: '[' 1 -eq 0 ']' 2024-09-13T12:26:23.660 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:124: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-13T12:26:25.973 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:26:25.985 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:127: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-09-13T12:26:26.010 INFO:tasks.workunit.client.0.smithi032.stdout:inferring bluefs devices from bluestore path 2024-09-13T12:26:28.571 INFO:tasks.workunit.client.0.smithi032.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-09-13T12:26:28.575 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:132: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-13T12:26:30.390 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:26:30.400 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:134: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --dev-target td/osd-bluefs-volume-ops/3/block.db --command bluefs-bdev-migrate 2024-09-13T12:26:30.425 INFO:tasks.workunit.client.0.smithi032.stdout:inferring bluefs devices from bluestore path 2024-09-13T12:26:32.501 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:139: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-13T12:26:34.313 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:26:34.325 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:141: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-09-13T12:26:34.325 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:26:34.325 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:26:34.325 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T12:26:34.325 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:26:34.326 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-13T12:26:34.326 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-13T12:26:34.326 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:26:34.326 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:26:34.326 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:26:34.326 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-13T12:26:34.326 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-13T12:26:34.326 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:26:34.326 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:26:34.326 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-13T12:26:34.326 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:26:34.326 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:26:34.327 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:26:34.327 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:26:34.327 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:26:34.327 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:26:34.327 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:26:34.327 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:26:34.327 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:26:34.327 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-13T12:26:34.327 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-13T12:26:34.327 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:26:34.327 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:26:34.327 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:26:34.327 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:26:34.328 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:26:34.328 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:26:34.328 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-13T12:26:34.328 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.0 2024-09-13T12:26:34.329 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T12:26:34.329 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:26:34.329 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-13T12:26:34.329 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T12:26:34.329 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:26:34.329 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:26:34.329 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:26:34.355 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:26:34.354+0000 7ff8a4816640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:26:34.358 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:26:34.356+0000 7ff8a4816640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:26:34.359 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:26:34.357+0000 7ff8a4816640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:26:34.779 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T12:26:34.779 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:26:34.779 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T12:26:34.779 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:26:34.779 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:26:34.780 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:26:34.780 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:26:34.781 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:26:34.781 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:26:34.781 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:26:35.230 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:26:35.708 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:26:35.706+0000 7ff8a4816640 -1 Falling back to public interface 2024-09-13T12:26:36.231 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:26:36.231 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:26:36.231 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:26:36.232 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:26:36.232 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:26:36.232 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:26:36.664 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:26:37.666 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:26:37.666 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:26:37.666 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:26:37.666 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:26:37.667 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:26:37.667 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:26:38.108 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:26:39.109 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:26:39.109 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:26:39.109 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:26:39.110 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:26:39.110 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:26:39.110 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:26:39.534 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:26:39.909 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:26:39.907+0000 7ff8a4816640 -1 osd.0 27 log_to_monitors true 2024-09-13T12:26:40.535 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:26:40.536 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:26:40.536 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:26:40.536 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:26:40.536 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:26:40.537 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:26:40.990 INFO:tasks.workunit.client.0.smithi032.stdout:osd.0 up in weight 1 up_from 38 up_thru 38 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6802/3122268130,v1:127.0.0.1:6803/3122268130] [v2:127.0.0.1:6804/3122268130,v1:127.0.0.1:6805/3122268130] exists,up f0200d83-d430-4fd5-a4ea-2304279eaa4f 2024-09-13T12:26:40.991 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:26:40.991 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:26:40.991 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:26:40.991 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-13T12:26:40.994 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: osd_pid0=125447 2024-09-13T12:26:40.994 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:143: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-09-13T12:26:40.994 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:26:40.995 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:26:40.995 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T12:26:40.995 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:26:40.995 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-13T12:26:40.995 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-13T12:26:40.995 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:26:40.995 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:26:40.995 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:26:40.995 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-13T12:26:40.995 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-13T12:26:40.995 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:26:40.995 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:26:40.995 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-13T12:26:40.996 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:26:40.996 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:26:40.996 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:26:40.996 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:26:40.996 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:26:40.996 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:26:40.996 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:26:40.996 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:26:40.996 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:26:40.996 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-13T12:26:40.996 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-13T12:26:40.997 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:26:40.997 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:26:40.997 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:26:40.997 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:26:40.997 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:26:40.997 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:26:40.997 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-13T12:26:40.998 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.1 2024-09-13T12:26:40.998 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T12:26:40.999 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:26:40.999 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-09-13T12:26:40.999 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T12:26:40.999 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:26:40.999 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:26:40.999 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:26:41.021 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:26:41.020+0000 7f5226158640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:26:41.023 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:26:41.022+0000 7f5226158640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:26:41.024 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:26:41.023+0000 7f5226158640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:26:41.435 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T12:26:41.435 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:26:41.436 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T12:26:41.436 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:26:41.436 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:26:41.436 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:26:41.436 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:26:41.436 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:26:41.436 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:26:41.437 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:26:41.893 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:26:42.618 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:26:42.617+0000 7f5226158640 -1 Falling back to public interface 2024-09-13T12:26:42.894 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:26:42.894 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:26:42.895 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:26:42.895 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:26:42.895 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:26:42.895 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:26:43.336 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:26:44.338 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:26:44.338 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:26:44.338 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:26:44.339 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:26:44.339 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:26:44.339 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:26:44.796 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:26:45.801 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:26:45.801 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:26:45.802 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:26:45.802 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:26:45.802 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:26:45.802 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:26:46.255 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:26:46.995 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:26:46.994+0000 7f5226158640 -1 osd.1 30 log_to_monitors true 2024-09-13T12:26:47.256 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:26:47.257 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:26:47.257 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:26:47.258 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:26:47.258 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:26:47.258 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:26:47.722 INFO:tasks.workunit.client.0.smithi032.stdout:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 31 last_clean_interval [10,30) [v2:127.0.0.1:6810/1895467681,v1:127.0.0.1:6811/1895467681] [v2:127.0.0.1:6812/1895467681,v1:127.0.0.1:6813/1895467681] exists,up 3bf12ce9-d5e2-4321-92bc-ee79874612e2 2024-09-13T12:26:47.723 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:26:47.723 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:26:47.723 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:26:47.723 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-13T12:26:47.726 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: osd_pid1=126420 2024-09-13T12:26:47.727 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:145: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-09-13T12:26:47.727 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:26:47.727 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:26:47.727 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-13T12:26:47.727 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:26:47.728 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-13T12:26:47.728 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-13T12:26:47.728 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:26:47.728 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:26:47.728 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:26:47.728 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-13T12:26:47.728 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-13T12:26:47.728 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:26:47.728 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:26:47.728 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-13T12:26:47.728 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:26:47.728 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:26:47.729 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:26:47.729 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:26:47.729 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:26:47.729 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:26:47.729 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:26:47.729 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:26:47.729 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:26:47.729 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-13T12:26:47.729 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-13T12:26:47.729 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:26:47.729 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:26:47.729 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:26:47.729 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:26:47.730 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:26:47.730 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:26:47.730 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-13T12:26:47.730 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.2 2024-09-13T12:26:47.731 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-13T12:26:47.731 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:26:47.731 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-09-13T12:26:47.732 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-13T12:26:47.732 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:26:47.732 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:26:47.732 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:26:47.754 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:26:47.752+0000 7f775a4f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:26:47.755 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:26:47.754+0000 7f775a4f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:26:47.757 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:26:47.755+0000 7f775a4f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:26:48.170 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-13T12:26:48.170 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:26:48.170 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T12:26:48.171 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:26:48.171 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:26:48.171 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:26:48.171 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:26:48.171 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:26:48.171 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:26:48.172 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:26:48.841 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:26:49.090 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:26:49.089+0000 7f775a4f1640 -1 Falling back to public interface 2024-09-13T12:26:49.842 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:26:49.843 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:26:49.843 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:26:49.843 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:26:49.843 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:26:49.844 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:26:50.284 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:26:51.286 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:26:51.286 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:26:51.286 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:26:51.287 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:26:51.287 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:26:51.287 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:26:51.719 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:26:52.274 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:26:52.273+0000 7f775a4f1640 -1 osd.2 33 log_to_monitors true 2024-09-13T12:26:52.721 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:26:52.721 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:26:52.721 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:26:52.722 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:26:52.722 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:26:52.722 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:26:53.487 INFO:tasks.workunit.client.0.smithi032.stdout:osd.2 up in weight 1 up_from 45 up_thru 31 down_at 34 last_clean_interval [15,33) [v2:127.0.0.1:6818/2792658033,v1:127.0.0.1:6819/2792658033] [v2:127.0.0.1:6820/2792658033,v1:127.0.0.1:6821/2792658033] exists,up 78431f38-564b-40c5-9ca6-e92448355c2a 2024-09-13T12:26:53.488 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:26:53.488 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:26:53.488 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:26:53.488 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-13T12:26:53.491 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: osd_pid2=127393 2024-09-13T12:26:53.491 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:147: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-09-13T12:26:53.491 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:26:53.491 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:26:53.491 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-13T12:26:53.491 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:26:53.491 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-13T12:26:53.492 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-13T12:26:53.492 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:26:53.492 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:26:53.492 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:26:53.492 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-13T12:26:53.492 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-13T12:26:53.492 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:26:53.492 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:26:53.493 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-13T12:26:53.493 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:26:53.493 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:26:53.493 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:26:53.493 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:26:53.493 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:26:53.493 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:26:53.493 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:26:53.493 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:26:53.493 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:26:53.493 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-13T12:26:53.493 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-13T12:26:53.493 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:26:53.494 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:26:53.494 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:26:53.494 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:26:53.494 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:26:53.494 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:26:53.494 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-13T12:26:53.494 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.3 2024-09-13T12:26:53.495 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-13T12:26:53.495 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:26:53.495 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-09-13T12:26:53.495 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-13T12:26:53.495 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:26:53.495 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:26:53.495 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:26:53.518 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:26:53.517+0000 7f0c4c3dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:26:53.520 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:26:53.520+0000 7f0c4c3dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:26:53.521 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:26:53.521+0000 7f0c4c3dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:26:54.052 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-13T12:26:54.052 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:26:54.052 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T12:26:54.052 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:26:54.052 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:26:54.052 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:26:54.053 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:26:54.053 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:26:54.053 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:26:54.053 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:26:54.529 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:26:55.136 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:26:55.135+0000 7f0c4c3dd640 -1 Falling back to public interface 2024-09-13T12:26:55.530 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:26:55.530 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:26:55.531 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:26:55.531 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:26:55.531 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:26:55.531 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:26:55.966 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:26:56.968 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:26:56.968 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:26:56.968 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:26:56.969 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:26:56.969 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:26:56.969 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:26:57.412 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:26:58.413 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:26:58.413 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:26:58.413 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:26:58.414 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:26:58.414 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:26:58.414 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:26:58.825 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:26:59.164 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:26:59.163+0000 7f0c4c3dd640 -1 osd.3 35 log_to_monitors true 2024-09-13T12:26:59.827 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:26:59.827 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:26:59.827 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:26:59.828 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:26:59.828 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:26:59.828 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:27:00.572 INFO:tasks.workunit.client.0.smithi032.stdout:osd.3 up in weight 1 up_from 49 up_thru 34 down_at 36 last_clean_interval [20,35) [v2:127.0.0.1:6826/2774756667,v1:127.0.0.1:6827/2774756667] [v2:127.0.0.1:6828/2774756667,v1:127.0.0.1:6829/2774756667] exists,up fc41ef3f-07fe-4ce3-aecb-2e8f6a9edb19 2024-09-13T12:27:00.573 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:27:00.573 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:27:00.573 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:27:00.573 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-13T12:27:00.574 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: osd_pid3=128336 2024-09-13T12:27:00.574 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:150: TEST_bluestore: wait_for_clean 2024-09-13T12:27:00.574 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T12:27:00.574 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T12:27:00.574 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T12:27:00.575 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T12:27:00.575 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T12:27:00.575 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T12:27:00.575 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T12:27:00.575 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T12:27:00.575 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T12:27:00.665 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T12:27:00.665 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T12:27:00.665 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T12:27:00.665 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T12:27:00.665 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T12:27:00.665 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T12:27:01.157 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T12:27:01.158 INFO:tasks.workunit.client.0.smithi032.stderr:1 2024-09-13T12:27:01.158 INFO:tasks.workunit.client.0.smithi032.stderr:2 2024-09-13T12:27:01.158 INFO:tasks.workunit.client.0.smithi032.stderr:3' 2024-09-13T12:27:01.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T12:27:01.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:27:01.158 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T12:27:01.430 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757255 2024-09-13T12:27:01.430 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757255 2024-09-13T12:27:01.430 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-163208757255' 2024-09-13T12:27:01.430 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:27:01.430 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T12:27:01.572 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=176093659141 2024-09-13T12:27:01.572 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 176093659141 2024-09-13T12:27:01.573 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-163208757255 1-176093659141' 2024-09-13T12:27:01.573 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:27:01.573 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T12:27:01.687 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528324 2024-09-13T12:27:01.687 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528324 2024-09-13T12:27:01.687 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-163208757255 1-176093659141 2-193273528324' 2024-09-13T12:27:01.687 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:27:01.687 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T12:27:01.803 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=210453397507 2024-09-13T12:27:01.804 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 210453397507 2024-09-13T12:27:01.804 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-163208757255 1-176093659141 2-193273528324 3-210453397507' 2024-09-13T12:27:01.804 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:27:01.804 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-163208757255 2024-09-13T12:27:01.804 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:27:01.804 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T12:27:01.805 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-163208757255 2024-09-13T12:27:01.805 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:27:01.806 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757255 2024-09-13T12:27:01.807 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.0 seq 163208757255 2024-09-13T12:27:01.808 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 163208757255' 2024-09-13T12:27:01.808 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:27:02.261 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757253 -lt 163208757255 2024-09-13T12:27:02.261 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T12:27:03.262 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T12:27:03.263 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:27:03.706 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757255 -lt 163208757255 2024-09-13T12:27:03.706 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:27:03.706 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-176093659141 2024-09-13T12:27:03.706 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:27:03.707 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T12:27:03.708 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-176093659141 2024-09-13T12:27:03.708 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:27:03.709 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=176093659141 2024-09-13T12:27:03.709 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 176093659141' 2024-09-13T12:27:03.709 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.1 seq 176093659141 2024-09-13T12:27:03.709 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T12:27:04.149 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659141 -lt 176093659141 2024-09-13T12:27:04.150 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:27:04.150 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-193273528324 2024-09-13T12:27:04.150 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:27:04.150 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T12:27:04.151 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-193273528324 2024-09-13T12:27:04.151 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:27:04.152 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528324 2024-09-13T12:27:04.152 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 193273528324' 2024-09-13T12:27:04.152 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.2 seq 193273528324 2024-09-13T12:27:04.152 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T12:27:04.642 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528325 -lt 193273528324 2024-09-13T12:27:04.642 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:27:04.642 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-210453397507 2024-09-13T12:27:04.643 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:27:04.643 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T12:27:04.644 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-210453397507 2024-09-13T12:27:04.644 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:27:04.645 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=210453397507 2024-09-13T12:27:04.645 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.3 seq 210453397507 2024-09-13T12:27:04.646 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 210453397507' 2024-09-13T12:27:04.646 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T12:27:05.068 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 210453397507 -lt 210453397507 2024-09-13T12:27:05.069 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T12:27:05.069 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:27:05.069 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:27:05.614 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 16 == 0 2024-09-13T12:27:05.614 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:27:05.615 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:27:05.615 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:27:05.615 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:27:05.616 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:27:05.616 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:27:05.616 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:27:06.015 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=16 2024-09-13T12:27:06.016 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:27:06.016 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:27:06.016 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:27:06.607 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 = 16 2024-09-13T12:27:06.607 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T12:27:06.607 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T12:27:06.607 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:153: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-13T12:27:06.652 INFO:tasks.workunit.client.0.smithi032.stdout:hints = 1 2024-09-13T12:27:06.652 INFO:tasks.workunit.client.0.smithi032.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-13T12:27:06.652 INFO:tasks.workunit.client.0.smithi032.stdout:Object prefix: benchmark_data_smithi032_129677 2024-09-13T12:27:06.652 INFO:tasks.workunit.client.0.smithi032.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T12:27:06.652 INFO:tasks.workunit.client.0.smithi032.stdout: 0 0 0 0 0 0 - 0 2024-09-13T12:27:07.652 INFO:tasks.workunit.client.0.smithi032.stdout: 1 16 76 60 0.23436 0.234375 0.259354 0.240876 2024-09-13T12:27:08.652 INFO:tasks.workunit.client.0.smithi032.stdout: 2 16 133 117 0.228488 0.222656 0.316897 0.251371 2024-09-13T12:27:09.652 INFO:tasks.workunit.client.0.smithi032.stdout: 3 16 190 174 0.226535 0.222656 0.377563 0.254399 2024-09-13T12:27:10.652 INFO:tasks.workunit.client.0.smithi032.stdout: 4 16 255 239 0.233369 0.253906 0.284625 0.262812 2024-09-13T12:27:11.652 INFO:tasks.workunit.client.0.smithi032.stdout: 5 16 310 294 0.229658 0.214844 0.327673 0.266748 2024-09-13T12:27:12.652 INFO:tasks.workunit.client.0.smithi032.stdout: 6 16 369 353 0.229787 0.230469 0.12119 0.263205 2024-09-13T12:27:13.652 INFO:tasks.workunit.client.0.smithi032.stdout: 7 16 437 421 0.234901 0.265625 0.204648 0.263468 2024-09-13T12:27:14.653 INFO:tasks.workunit.client.0.smithi032.stdout: 8 16 495 479 0.233856 0.226562 0.2741 0.264944 2024-09-13T12:27:15.653 INFO:tasks.workunit.client.0.smithi032.stdout: 9 16 545 529 0.229571 0.195312 0.341989 0.266382 2024-09-13T12:27:16.653 INFO:tasks.workunit.client.0.smithi032.stdout: 10 16 596 580 0.226533 0.199219 0.162239 0.271436 2024-09-13T12:27:17.654 INFO:tasks.workunit.client.0.smithi032.stdout: 11 16 659 643 0.228308 0.246094 0.211676 0.270739 2024-09-13T12:27:18.653 INFO:tasks.workunit.client.0.smithi032.stdout: 12 16 715 699 0.227509 0.21875 0.211654 0.269751 2024-09-13T12:27:19.653 INFO:tasks.workunit.client.0.smithi032.stdout: 13 16 781 765 0.229838 0.257812 0.175594 0.269535 2024-09-13T12:27:20.653 INFO:tasks.workunit.client.0.smithi032.stdout: 14 16 836 820 0.228764 0.214844 0.359801 0.271127 2024-09-13T12:27:21.653 INFO:tasks.workunit.client.0.smithi032.stdout: 15 16 883 867 0.225751 0.183594 0.183344 0.27248 2024-09-13T12:27:22.654 INFO:tasks.workunit.client.0.smithi032.stdout: 16 16 941 925 0.225801 0.226562 0.129351 0.273437 2024-09-13T12:27:23.654 INFO:tasks.workunit.client.0.smithi032.stdout: 17 16 1010 994 0.228371 0.269531 0.158028 0.271544 2024-09-13T12:27:24.654 INFO:tasks.workunit.client.0.smithi032.stdout: 18 16 1091 1075 0.23326 0.316406 0.226677 0.26713 2024-09-13T12:27:25.654 INFO:tasks.workunit.client.0.smithi032.stdout: 19 16 1156 1140 0.234344 0.253906 0.221818 0.265663 2024-09-13T12:27:26.654 INFO:tasks.workunit.client.0.smithi032.stdout:2024-09-13T12:27:26.654509+0000 min lat: 0.0883711 max lat: 0.450895 avg lat: 0.264677 2024-09-13T12:27:26.654 INFO:tasks.workunit.client.0.smithi032.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T12:27:26.654 INFO:tasks.workunit.client.0.smithi032.stdout: 20 16 1217 1201 0.23454 0.238281 0.1839 0.264677 2024-09-13T12:27:27.654 INFO:tasks.workunit.client.0.smithi032.stdout: 21 16 1300 1284 0.238807 0.324219 0.172517 0.260393 2024-09-13T12:27:28.655 INFO:tasks.workunit.client.0.smithi032.stdout: 22 16 1382 1366 0.242511 0.320312 0.175768 0.257097 2024-09-13T12:27:29.655 INFO:tasks.workunit.client.0.smithi032.stdout: 23 16 1462 1446 0.245552 0.3125 0.218183 0.253082 2024-09-13T12:27:30.655 INFO:tasks.workunit.client.0.smithi032.stdout: 24 16 1550 1534 0.249641 0.34375 0.189583 0.249601 2024-09-13T12:27:31.655 INFO:tasks.workunit.client.0.smithi032.stdout: 25 16 1630 1614 0.252154 0.3125 0.209309 0.246756 2024-09-13T12:27:32.655 INFO:tasks.workunit.client.0.smithi032.stdout: 26 16 1712 1696 0.254774 0.320312 0.200852 0.244035 2024-09-13T12:27:33.655 INFO:tasks.workunit.client.0.smithi032.stdout: 27 16 1791 1775 0.256766 0.308594 0.1233 0.24269 2024-09-13T12:27:34.655 INFO:tasks.workunit.client.0.smithi032.stdout: 28 16 1860 1844 0.257221 0.269531 0.218667 0.24193 2024-09-13T12:27:35.655 INFO:tasks.workunit.client.0.smithi032.stdout: 29 16 1937 1921 0.258721 0.300781 0.185541 0.241142 2024-09-13T12:27:36.656 INFO:tasks.workunit.client.0.smithi032.stdout: 30 16 2006 1990 0.25908 0.269531 0.243203 0.240322 2024-09-13T12:27:37.657 INFO:tasks.workunit.client.0.smithi032.stdout:Total time run: 30.2523 2024-09-13T12:27:37.657 INFO:tasks.workunit.client.0.smithi032.stdout:Total writes made: 2006 2024-09-13T12:27:37.657 INFO:tasks.workunit.client.0.smithi032.stdout:Write size: 4096 2024-09-13T12:27:37.657 INFO:tasks.workunit.client.0.smithi032.stdout:Object size: 4096 2024-09-13T12:27:37.657 INFO:tasks.workunit.client.0.smithi032.stdout:Bandwidth (MB/sec): 0.259019 2024-09-13T12:27:37.657 INFO:tasks.workunit.client.0.smithi032.stdout:Stddev Bandwidth: 0.0448012 2024-09-13T12:27:37.657 INFO:tasks.workunit.client.0.smithi032.stdout:Max bandwidth (MB/sec): 0.34375 2024-09-13T12:27:37.657 INFO:tasks.workunit.client.0.smithi032.stdout:Min bandwidth (MB/sec): 0.183594 2024-09-13T12:27:37.657 INFO:tasks.workunit.client.0.smithi032.stdout:Average IOPS: 66 2024-09-13T12:27:37.657 INFO:tasks.workunit.client.0.smithi032.stdout:Stddev IOPS: 11.4691 2024-09-13T12:27:37.657 INFO:tasks.workunit.client.0.smithi032.stdout:Max IOPS: 88 2024-09-13T12:27:37.658 INFO:tasks.workunit.client.0.smithi032.stdout:Min IOPS: 47 2024-09-13T12:27:37.658 INFO:tasks.workunit.client.0.smithi032.stdout:Average Latency(s): 0.240402 2024-09-13T12:27:37.658 INFO:tasks.workunit.client.0.smithi032.stdout:Stddev Latency(s): 0.065718 2024-09-13T12:27:37.658 INFO:tasks.workunit.client.0.smithi032.stdout:Max latency(s): 0.450895 2024-09-13T12:27:37.658 INFO:tasks.workunit.client.0.smithi032.stdout:Min latency(s): 0.0883711 2024-09-13T12:27:37.742 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 125447 2024-09-13T12:27:37.743 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: sleep 1 2024-09-13T12:27:38.744 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 125447 2024-09-13T12:27:38.744 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 156: kill: (125447) - No such process 2024-09-13T12:27:38.744 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:157: TEST_bluestore: ceph osd down 0 2024-09-13T12:27:39.324 INFO:tasks.workunit.client.0.smithi032.stderr:osd.0 is already down. 2024-09-13T12:27:39.335 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 126420 2024-09-13T12:27:39.335 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: sleep 1 2024-09-13T12:27:40.337 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 126420 2024-09-13T12:27:40.337 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 158: kill: (126420) - No such process 2024-09-13T12:27:40.338 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:159: TEST_bluestore: ceph osd down 1 2024-09-13T12:27:40.770 INFO:tasks.workunit.client.0.smithi032.stderr:osd.1 is already down. 2024-09-13T12:27:40.781 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 127393 2024-09-13T12:27:40.782 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: sleep 1 2024-09-13T12:27:41.782 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 127393 2024-09-13T12:27:41.783 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 160: kill: (127393) - No such process 2024-09-13T12:27:41.783 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:161: TEST_bluestore: ceph osd down 2 2024-09-13T12:27:42.199 INFO:tasks.workunit.client.0.smithi032.stderr:osd.2 is already down. 2024-09-13T12:27:42.210 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 128336 2024-09-13T12:27:42.210 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: sleep 1 2024-09-13T12:27:43.212 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 128336 2024-09-13T12:27:43.212 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 162: kill: (128336) - No such process 2024-09-13T12:27:43.213 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:163: TEST_bluestore: ceph osd down 3 2024-09-13T12:27:43.734 INFO:tasks.workunit.client.0.smithi032.stderr:osd.3 is already down. 2024-09-13T12:27:43.746 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:166: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-13T12:27:45.557 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:27:45.567 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:168: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal count=512 bs=1M 2024-09-13T12:27:45.831 INFO:tasks.workunit.client.0.smithi032.stderr:512+0 records in 2024-09-13T12:27:45.831 INFO:tasks.workunit.client.0.smithi032.stderr:512+0 records out 2024-09-13T12:27:45.832 INFO:tasks.workunit.client.0.smithi032.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.262999 s, 2.0 GB/s 2024-09-13T12:27:45.832 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:169: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --dev-target td/osd-bluefs-volume-ops/0/wal --command bluefs-bdev-new-wal 2024-09-13T12:27:45.852 INFO:tasks.workunit.client.0.smithi032.stdout:inferring bluefs devices from bluestore path 2024-09-13T12:27:51.406 INFO:tasks.workunit.client.0.smithi032.stdout:WAL device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal 2024-09-13T12:27:51.410 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:173: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-13T12:27:53.472 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:27:53.482 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:176: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-13T12:27:55.295 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:27:55.306 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:178: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db count=1024 bs=1M 2024-09-13T12:27:55.817 INFO:tasks.workunit.client.0.smithi032.stderr:1024+0 records in 2024-09-13T12:27:55.817 INFO:tasks.workunit.client.0.smithi032.stderr:1024+0 records out 2024-09-13T12:27:55.817 INFO:tasks.workunit.client.0.smithi032.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.50952 s, 2.1 GB/s 2024-09-13T12:27:55.817 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:179: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --dev-target td/osd-bluefs-volume-ops/1/db --command bluefs-bdev-new-db 2024-09-13T12:27:55.837 INFO:tasks.workunit.client.0.smithi032.stdout:inferring bluefs devices from bluestore path 2024-09-13T12:28:04.393 INFO:tasks.workunit.client.0.smithi032.stdout:DB device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db 2024-09-13T12:28:04.396 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:183: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block --dev-target td/osd-bluefs-volume-ops/1/block.db --command bluefs-bdev-migrate 2024-09-13T12:28:04.419 INFO:tasks.workunit.client.0.smithi032.stdout:inferring bluefs devices from bluestore path 2024-09-13T12:28:05.470 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:05.469+0000 7f985b31c980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:28:06.241 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:06.239+0000 7f985b31c980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:28:07.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:188: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-13T12:28:07.793 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:07.791+0000 7fe1ef31c980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:28:08.556 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:08.554+0000 7fe1ef31c980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:28:09.551 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:28:09.558 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:191: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-13T12:28:10.877 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:28:10.883 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:193: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-db 2024-09-13T12:28:10.906 INFO:tasks.workunit.client.0.smithi032.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-09-13T12:28:10.906 INFO:tasks.workunit.client.0.smithi032.stdout:inferring bluefs devices from bluestore path 2024-09-13T12:28:12.191 INFO:tasks.workunit.client.0.smithi032.stdout:DB device added 2024-09-13T12:28:12.194 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:196: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-wal 2024-09-13T12:28:12.219 INFO:tasks.workunit.client.0.smithi032.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-09-13T12:28:12.219 INFO:tasks.workunit.client.0.smithi032.stdout:inferring bluefs devices from bluestore path 2024-09-13T12:28:13.013 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:13.010+0000 7f066f51d980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:28:13.524 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:13.522+0000 7f066f51d980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:28:13.577 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:13.576+0000 7f066f51d980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:28:14.272 INFO:tasks.workunit.client.0.smithi032.stdout:WAL device added 2024-09-13T12:28:14.276 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:199: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/block.db --command bluefs-bdev-migrate 2024-09-13T12:28:14.299 INFO:tasks.workunit.client.0.smithi032.stdout:inferring bluefs devices from bluestore path 2024-09-13T12:28:15.369 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:15.368+0000 7fdfe24d0980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:28:16.136 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:16.135+0000 7fdfe24d0980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:28:17.383 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:204: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-13T12:28:17.943 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:17.943+0000 7fb893d21980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:28:18.709 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:18.708+0000 7fb893d21980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:28:19.704 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:28:19.711 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:207: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-13T12:28:21.531 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:28:21.539 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:209: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --command bluefs-bdev-new-wal 2024-09-13T12:28:21.564 INFO:tasks.workunit.client.0.smithi032.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-09-13T12:28:21.564 INFO:tasks.workunit.client.0.smithi032.stdout:inferring bluefs devices from bluestore path 2024-09-13T12:28:23.618 INFO:tasks.workunit.client.0.smithi032.stdout:WAL device added 2024-09-13T12:28:23.622 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:212: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-09-13T12:28:23.645 INFO:tasks.workunit.client.0.smithi032.stdout:inferring bluefs devices from bluestore path 2024-09-13T12:28:26.711 INFO:tasks.workunit.client.0.smithi032.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block.db 2024-09-13T12:28:26.716 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:217: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-13T12:28:28.527 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:28:28.537 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:219: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-09-13T12:28:28.537 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:28:28.538 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:28:28.538 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T12:28:28.538 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:28:28.538 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-13T12:28:28.538 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-13T12:28:28.538 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:28:28.538 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:28:28.538 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:28:28.538 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-13T12:28:28.538 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-13T12:28:28.538 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:28:28.538 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:28:28.539 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-13T12:28:28.539 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:28:28.539 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:28:28.539 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:28:28.539 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:28:28.539 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:28:28.539 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:28:28.539 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:28:28.539 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:28:28.539 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:28:28.539 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-13T12:28:28.539 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-13T12:28:28.540 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:28:28.540 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:28:28.540 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:28:28.540 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:28:28.540 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:28:28.540 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:28:28.540 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-13T12:28:28.540 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.0 2024-09-13T12:28:28.542 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T12:28:28.542 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:28:28.542 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-13T12:28:28.542 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T12:28:28.542 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:28:28.543 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:28:28.543 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:28:28.568 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:28.567+0000 7f4a62416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:28:28.570 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:28.570+0000 7f4a62416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:28:28.571 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:28.571+0000 7f4a62416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:28:28.979 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T12:28:28.979 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:28:28.979 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T12:28:28.979 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:28:28.979 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:28:28.979 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:28:28.980 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:28:28.980 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:28:28.980 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:28:28.980 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:28:29.413 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:28:30.167 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:30.166+0000 7f4a62416640 -1 Falling back to public interface 2024-09-13T12:28:30.414 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:28:30.414 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:28:30.415 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:28:30.415 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:28:30.415 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:28:30.415 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:28:30.855 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:28:31.857 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:28:31.857 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:28:31.857 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:28:31.858 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:28:31.858 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:28:31.858 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:28:32.292 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:28:33.293 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:28:33.293 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:28:33.293 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:28:33.294 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:28:33.294 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:28:33.294 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:28:33.727 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:28:34.728 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:28:34.728 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:28:34.729 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:28:34.729 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:28:34.729 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:28:34.730 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:28:34.771 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:34.770+0000 7f4a62416640 -1 osd.0 51 log_to_monitors true 2024-09-13T12:28:35.149 INFO:tasks.workunit.client.0.smithi032.stdout:osd.0 up in weight 1 up_from 61 up_thru 49 down_at 52 last_clean_interval [38,51) [v2:127.0.0.1:6802/2681595223,v1:127.0.0.1:6803/2681595223] [v2:127.0.0.1:6804/2681595223,v1:127.0.0.1:6805/2681595223] exists,up f0200d83-d430-4fd5-a4ea-2304279eaa4f 2024-09-13T12:28:35.149 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:28:35.150 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:28:35.150 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:28:35.150 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-13T12:28:35.151 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: osd_pid0=135828 2024-09-13T12:28:35.151 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:221: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-09-13T12:28:35.151 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:28:35.151 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:28:35.151 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T12:28:35.151 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:28:35.151 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-13T12:28:35.151 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-13T12:28:35.154 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:28:35.154 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:28:35.154 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:28:35.154 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-13T12:28:35.155 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-13T12:28:35.155 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:28:35.155 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:28:35.157 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-13T12:28:35.157 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:28:35.157 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:28:35.157 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:28:35.157 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:28:35.157 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:28:35.158 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:28:35.158 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:28:35.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:28:35.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:28:35.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-13T12:28:35.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-13T12:28:35.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:28:35.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:28:35.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:28:35.158 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:28:35.159 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:28:35.159 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:28:35.159 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-13T12:28:35.159 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.1 2024-09-13T12:28:35.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T12:28:35.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:28:35.160 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-09-13T12:28:35.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T12:28:35.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:28:35.160 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:28:35.161 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:28:35.179 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:35.178+0000 7f50ae416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:28:35.244 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:35.242+0000 7f50ae416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:28:35.246 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:35.243+0000 7f50ae416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:28:35.575 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T12:28:35.576 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:28:35.576 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T12:28:35.576 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:28:35.576 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:28:35.576 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:28:35.576 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:28:35.577 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:28:35.577 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:28:35.577 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:28:36.001 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:28:36.840 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:36.839+0000 7f50ae416640 -1 Falling back to public interface 2024-09-13T12:28:37.003 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:28:37.003 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:28:37.003 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:28:37.004 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:28:37.004 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:28:37.004 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:28:37.487 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:28:38.488 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:28:38.488 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:28:38.488 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:28:38.489 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:28:38.489 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:28:38.489 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:28:38.489 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:38.489+0000 7f50ae416640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:28:38.925 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:28:39.252 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:39.251+0000 7f50ae416640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:28:39.922 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:28:39.922 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:28:39.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:28:39.923 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:28:39.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:28:39.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:28:40.276 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:40.275+0000 7f50ae416640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:28:40.364 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:28:41.033 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:41.032+0000 7f50ae416640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:28:41.366 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:28:41.366 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:28:41.366 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:28:41.367 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:28:41.367 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:28:41.368 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:28:41.415 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:41.415+0000 7f50ae416640 -1 osd.1 54 log_to_monitors true 2024-09-13T12:28:41.791 INFO:tasks.workunit.client.0.smithi032.stdout:osd.1 up in weight 1 up_from 64 up_thru 52 down_at 55 last_clean_interval [41,54) [v2:127.0.0.1:6810/1729432583,v1:127.0.0.1:6811/1729432583] [v2:127.0.0.1:6812/1729432583,v1:127.0.0.1:6813/1729432583] exists,up 3bf12ce9-d5e2-4321-92bc-ee79874612e2 2024-09-13T12:28:41.795 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:28:41.795 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:28:41.795 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:28:41.795 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-13T12:28:41.795 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: osd_pid1=136824 2024-09-13T12:28:41.796 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:223: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-09-13T12:28:41.796 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:28:41.796 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:28:41.796 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-13T12:28:41.796 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:28:41.796 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-13T12:28:41.796 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-13T12:28:41.796 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:28:41.796 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:28:41.796 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:28:41.796 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-13T12:28:41.796 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-13T12:28:41.796 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:28:41.797 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:28:41.797 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-13T12:28:41.797 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:28:41.797 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:28:41.797 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:28:41.797 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:28:41.797 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:28:41.797 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:28:41.797 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:28:41.797 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:28:41.797 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:28:41.797 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-13T12:28:41.797 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-13T12:28:41.797 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:28:41.797 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:28:41.798 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:28:41.798 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:28:41.798 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:28:41.798 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:28:41.798 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-13T12:28:41.798 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.2 2024-09-13T12:28:41.799 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-13T12:28:41.799 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:28:41.799 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-09-13T12:28:41.799 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-13T12:28:41.799 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:28:41.799 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:28:41.799 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:28:41.822 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:41.822+0000 7f05f65a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:28:41.824 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:41.824+0000 7f05f65a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:28:41.825 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:41.825+0000 7f05f65a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:28:42.260 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-13T12:28:42.260 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:28:42.260 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T12:28:42.260 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:28:42.260 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:28:42.260 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:28:42.260 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:28:42.260 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:28:42.261 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:28:42.261 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:28:42.703 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:28:43.668 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:43.667+0000 7f05f65a8640 -1 Falling back to public interface 2024-09-13T12:28:43.704 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:28:43.704 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:28:43.704 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:28:43.705 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:28:43.705 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:28:43.705 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:28:44.147 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:28:45.149 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:28:45.149 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:28:45.149 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:28:45.150 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:28:45.150 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:28:45.150 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:28:45.582 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:28:45.602 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:45.601+0000 7f05f65a8640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:28:46.365 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:46.364+0000 7f05f65a8640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:28:46.583 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:28:46.583 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:28:46.584 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:28:46.584 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:28:46.584 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:28:46.584 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:28:47.009 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:28:47.371 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:47.370+0000 7f05f65a8640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:28:48.010 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:28:48.010 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:28:48.010 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:28:48.010 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:28:48.011 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:28:48.011 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:28:48.143 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:48.142+0000 7f05f65a8640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:28:48.433 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:28:49.263 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:49.263+0000 7f05f65a8640 -1 osd.2 56 log_to_monitors true 2024-09-13T12:28:49.435 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:28:49.435 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:28:49.435 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T12:28:49.435 INFO:tasks.workunit.client.0.smithi032.stdout:5 2024-09-13T12:28:49.436 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:28:49.436 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:28:50.016 INFO:tasks.workunit.client.0.smithi032.stdout:osd.2 up in weight 1 up_from 68 up_thru 55 down_at 57 last_clean_interval [45,56) [v2:127.0.0.1:6818/1556147913,v1:127.0.0.1:6819/1556147913] [v2:127.0.0.1:6820/1556147913,v1:127.0.0.1:6821/1556147913] exists,up 78431f38-564b-40c5-9ca6-e92448355c2a 2024-09-13T12:28:50.017 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:28:50.017 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:28:50.017 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:28:50.017 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-13T12:28:50.020 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: osd_pid2=137785 2024-09-13T12:28:50.020 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:225: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-09-13T12:28:50.020 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:28:50.020 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:28:50.021 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-13T12:28:50.021 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:28:50.021 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-13T12:28:50.021 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-13T12:28:50.021 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:28:50.021 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:28:50.021 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:28:50.021 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-13T12:28:50.022 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-13T12:28:50.022 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:28:50.022 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:28:50.022 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-13T12:28:50.022 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:28:50.022 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:28:50.022 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:28:50.022 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:28:50.023 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:28:50.023 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:28:50.023 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:28:50.023 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:28:50.023 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:28:50.023 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-13T12:28:50.023 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-13T12:28:50.023 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:28:50.023 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:28:50.023 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:28:50.023 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:28:50.023 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:28:50.023 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:28:50.023 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-13T12:28:50.024 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.3 2024-09-13T12:28:50.025 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-13T12:28:50.025 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:28:50.025 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-09-13T12:28:50.025 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-13T12:28:50.025 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:28:50.025 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:28:50.025 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:28:50.047 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:50.045+0000 7f1ddd345640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:28:50.049 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:50.047+0000 7f1ddd345640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:28:50.050 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:50.049+0000 7f1ddd345640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:28:50.711 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-13T12:28:50.712 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:28:50.712 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T12:28:50.712 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:28:50.712 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:28:50.712 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:28:50.712 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:28:50.712 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:28:50.713 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:28:50.713 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:28:51.278 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:28:51.403 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:51.401+0000 7f1ddd345640 -1 Falling back to public interface 2024-09-13T12:28:52.279 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:28:52.279 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:28:52.279 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:28:52.280 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:28:52.280 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:28:52.280 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:28:52.726 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:28:53.728 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:28:53.728 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:28:53.728 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:28:53.728 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:28:53.729 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:28:53.729 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:28:54.170 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:28:55.171 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:28:55.172 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:28:55.172 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:28:55.172 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:28:55.172 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:28:55.173 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:28:55.591 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:28:56.128 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:28:56.126+0000 7f1ddd345640 -1 osd.3 58 log_to_monitors true 2024-09-13T12:28:56.593 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:28:56.593 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:28:56.593 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:28:56.593 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:28:56.594 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:28:56.594 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:28:57.111 INFO:tasks.workunit.client.0.smithi032.stdout:osd.3 up in weight 1 up_from 73 up_thru 57 down_at 59 last_clean_interval [49,58) [v2:127.0.0.1:6826/3599744190,v1:127.0.0.1:6827/3599744190] [v2:127.0.0.1:6828/3599744190,v1:127.0.0.1:6829/3599744190] exists,up fc41ef3f-07fe-4ce3-aecb-2e8f6a9edb19 2024-09-13T12:28:57.112 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:28:57.112 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:28:57.112 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:28:57.112 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-13T12:28:57.112 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: osd_pid3=138788 2024-09-13T12:28:57.113 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:229: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-13T12:28:57.144 INFO:tasks.workunit.client.0.smithi032.stdout:hints = 1 2024-09-13T12:28:57.144 INFO:tasks.workunit.client.0.smithi032.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-13T12:28:57.144 INFO:tasks.workunit.client.0.smithi032.stdout:Object prefix: benchmark_data_smithi032_139746 2024-09-13T12:28:57.144 INFO:tasks.workunit.client.0.smithi032.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T12:28:57.144 INFO:tasks.workunit.client.0.smithi032.stdout: 0 0 0 0 0 0 - 0 2024-09-13T12:28:58.144 INFO:tasks.workunit.client.0.smithi032.stdout: 1 16 19 3 0.0117172 0.0117188 0.246088 0.27297 2024-09-13T12:28:59.144 INFO:tasks.workunit.client.0.smithi032.stdout: 2 16 56 40 0.0781125 0.144531 0.201448 0.734585 2024-09-13T12:29:00.144 INFO:tasks.workunit.client.0.smithi032.stdout: 3 16 107 91 0.118473 0.199219 0.195532 0.489744 2024-09-13T12:29:01.145 INFO:tasks.workunit.client.0.smithi032.stdout: 4 16 154 138 0.134747 0.183594 0.354656 0.440644 2024-09-13T12:29:02.145 INFO:tasks.workunit.client.0.smithi032.stdout: 5 16 205 189 0.147636 0.199219 0.359493 0.402936 2024-09-13T12:29:03.145 INFO:tasks.workunit.client.0.smithi032.stdout: 6 16 260 244 0.158832 0.214844 0.237019 0.385295 2024-09-13T12:29:04.145 INFO:tasks.workunit.client.0.smithi032.stdout: 7 16 309 293 0.163481 0.191406 0.300993 0.372115 2024-09-13T12:29:05.145 INFO:tasks.workunit.client.0.smithi032.stdout: 8 16 366 350 0.170874 0.222656 0.187481 0.361296 2024-09-13T12:29:06.145 INFO:tasks.workunit.client.0.smithi032.stdout: 9 16 410 394 0.170982 0.171875 0.327364 0.359805 2024-09-13T12:29:07.146 INFO:tasks.workunit.client.0.smithi032.stdout: 10 16 456 440 0.17185 0.179688 0.355158 0.356641 2024-09-13T12:29:08.146 INFO:tasks.workunit.client.0.smithi032.stdout: 11 16 502 486 0.172559 0.179688 0.351539 0.35434 2024-09-13T12:29:09.146 INFO:tasks.workunit.client.0.smithi032.stdout: 12 16 549 533 0.173477 0.183594 0.337752 0.354363 2024-09-13T12:29:10.146 INFO:tasks.workunit.client.0.smithi032.stdout: 13 16 596 580 0.174253 0.183594 0.335689 0.352032 2024-09-13T12:29:11.146 INFO:tasks.workunit.client.0.smithi032.stdout: 14 16 648 632 0.176314 0.203125 0.209085 0.349256 2024-09-13T12:29:12.146 INFO:tasks.workunit.client.0.smithi032.stdout: 15 16 701 685 0.17836 0.207031 0.317649 0.346441 2024-09-13T12:29:13.146 INFO:tasks.workunit.client.0.smithi032.stdout: 16 16 750 734 0.179173 0.191406 0.314995 0.344684 2024-09-13T12:29:14.146 INFO:tasks.workunit.client.0.smithi032.stdout: 17 16 806 790 0.1815 0.21875 0.335592 0.340937 2024-09-13T12:29:15.146 INFO:tasks.workunit.client.0.smithi032.stdout: 18 16 857 841 0.182483 0.199219 0.272526 0.339572 2024-09-13T12:29:16.147 INFO:tasks.workunit.client.0.smithi032.stdout: 19 16 898 882 0.181307 0.160156 0.406821 0.339254 2024-09-13T12:29:17.147 INFO:tasks.workunit.client.0.smithi032.stdout:2024-09-13T12:29:17.147214+0000 min lat: 0.173317 max lat: 1.58427 avg lat: 0.341399 2024-09-13T12:29:17.147 INFO:tasks.workunit.client.0.smithi032.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T12:29:17.147 INFO:tasks.workunit.client.0.smithi032.stdout: 20 16 948 932 0.182006 0.195312 0.234403 0.341399 2024-09-13T12:29:18.147 INFO:tasks.workunit.client.0.smithi032.stdout: 21 16 992 976 0.181522 0.171875 0.375018 0.341004 2024-09-13T12:29:19.148 INFO:tasks.workunit.client.0.smithi032.stdout: 22 16 1038 1022 0.181437 0.179688 0.36383 0.341684 2024-09-13T12:29:20.147 INFO:tasks.workunit.client.0.smithi032.stdout: 23 16 1084 1068 0.18136 0.179688 0.39206 0.342055 2024-09-13T12:29:21.147 INFO:tasks.workunit.client.0.smithi032.stdout: 24 16 1128 1112 0.180964 0.171875 0.214146 0.343241 2024-09-13T12:29:22.147 INFO:tasks.workunit.client.0.smithi032.stdout: 25 16 1170 1154 0.180287 0.164062 0.350417 0.344149 2024-09-13T12:29:23.148 INFO:tasks.workunit.client.0.smithi032.stdout: 26 16 1221 1205 0.181014 0.199219 0.225291 0.343825 2024-09-13T12:29:24.148 INFO:tasks.workunit.client.0.smithi032.stdout: 27 16 1270 1254 0.181398 0.191406 0.341919 0.342482 2024-09-13T12:29:25.148 INFO:tasks.workunit.client.0.smithi032.stdout: 28 16 1317 1301 0.181476 0.183594 0.33664 0.341776 2024-09-13T12:29:26.148 INFO:tasks.workunit.client.0.smithi032.stdout: 29 16 1364 1348 0.181548 0.183594 0.243169 0.342159 2024-09-13T12:29:27.148 INFO:tasks.workunit.client.0.smithi032.stdout: 30 16 1414 1398 0.182006 0.195312 0.324424 0.341797 2024-09-13T12:29:28.149 INFO:tasks.workunit.client.0.smithi032.stdout:Total time run: 30.2819 2024-09-13T12:29:28.149 INFO:tasks.workunit.client.0.smithi032.stdout:Total writes made: 1414 2024-09-13T12:29:28.149 INFO:tasks.workunit.client.0.smithi032.stdout:Write size: 4096 2024-09-13T12:29:28.149 INFO:tasks.workunit.client.0.smithi032.stdout:Object size: 4096 2024-09-13T12:29:28.149 INFO:tasks.workunit.client.0.smithi032.stdout:Bandwidth (MB/sec): 0.1824 2024-09-13T12:29:28.149 INFO:tasks.workunit.client.0.smithi032.stdout:Stddev Bandwidth: 0.0364365 2024-09-13T12:29:28.149 INFO:tasks.workunit.client.0.smithi032.stdout:Max bandwidth (MB/sec): 0.222656 2024-09-13T12:29:28.149 INFO:tasks.workunit.client.0.smithi032.stdout:Min bandwidth (MB/sec): 0.0117188 2024-09-13T12:29:28.149 INFO:tasks.workunit.client.0.smithi032.stdout:Average IOPS: 46 2024-09-13T12:29:28.149 INFO:tasks.workunit.client.0.smithi032.stdout:Stddev IOPS: 9.32775 2024-09-13T12:29:28.149 INFO:tasks.workunit.client.0.smithi032.stdout:Max IOPS: 57 2024-09-13T12:29:28.149 INFO:tasks.workunit.client.0.smithi032.stdout:Min IOPS: 3 2024-09-13T12:29:28.150 INFO:tasks.workunit.client.0.smithi032.stdout:Average Latency(s): 0.341672 2024-09-13T12:29:28.150 INFO:tasks.workunit.client.0.smithi032.stdout:Stddev Latency(s): 0.13068 2024-09-13T12:29:28.150 INFO:tasks.workunit.client.0.smithi032.stdout:Max latency(s): 1.58427 2024-09-13T12:29:28.150 INFO:tasks.workunit.client.0.smithi032.stdout:Min latency(s): 0.173317 2024-09-13T12:29:28.288 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 135828 2024-09-13T12:29:28.288 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: sleep 1 2024-09-13T12:29:29.290 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 135828 2024-09-13T12:29:29.290 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 232: kill: (135828) - No such process 2024-09-13T12:29:29.290 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:233: TEST_bluestore: ceph osd down 0 2024-09-13T12:29:29.707 INFO:tasks.workunit.client.0.smithi032.stderr:osd.0 is already down. 2024-09-13T12:29:29.719 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 136824 2024-09-13T12:29:29.719 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: sleep 1 2024-09-13T12:29:30.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 136824 2024-09-13T12:29:30.721 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 234: kill: (136824) - No such process 2024-09-13T12:29:30.721 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:235: TEST_bluestore: ceph osd down 1 2024-09-13T12:29:31.232 INFO:tasks.workunit.client.0.smithi032.stderr:osd.1 is already down. 2024-09-13T12:29:31.243 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 137785 2024-09-13T12:29:31.244 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: sleep 1 2024-09-13T12:29:32.245 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 137785 2024-09-13T12:29:32.245 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 236: kill: (137785) - No such process 2024-09-13T12:29:32.245 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:237: TEST_bluestore: ceph osd down 2 2024-09-13T12:29:32.711 INFO:tasks.workunit.client.0.smithi032.stderr:osd.2 is already down. 2024-09-13T12:29:32.723 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 138788 2024-09-13T12:29:32.723 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: sleep 1 2024-09-13T12:29:33.724 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 138788 2024-09-13T12:29:33.724 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 238: kill: (138788) - No such process 2024-09-13T12:29:33.724 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:239: TEST_bluestore: ceph osd down 3 2024-09-13T12:29:34.262 INFO:tasks.workunit.client.0.smithi032.stderr:osd.3 is already down. 2024-09-13T12:29:34.273 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:242: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-13T12:29:36.581 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:29:36.593 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:244: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/db2 count=1024 bs=1M 2024-09-13T12:29:37.103 INFO:tasks.workunit.client.0.smithi032.stderr:1024+0 records in 2024-09-13T12:29:37.103 INFO:tasks.workunit.client.0.smithi032.stderr:1024+0 records out 2024-09-13T12:29:37.103 INFO:tasks.workunit.client.0.smithi032.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.508556 s, 2.1 GB/s 2024-09-13T12:29:37.103 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:245: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/db2 --command bluefs-bdev-migrate 2024-09-13T12:29:37.123 INFO:tasks.workunit.client.0.smithi032.stdout:inferring bluefs devices from bluestore path 2024-09-13T12:29:46.689 INFO:tasks.workunit.client.0.smithi032.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-09-13T12:29:46.690 INFO:tasks.workunit.client.0.smithi032.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/db2 2024-09-13T12:29:46.691 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:250: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-13T12:29:49.005 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:29:49.012 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:254: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal2 count=512 bs=1M 2024-09-13T12:29:49.277 INFO:tasks.workunit.client.0.smithi032.stderr:512+0 records in 2024-09-13T12:29:49.277 INFO:tasks.workunit.client.0.smithi032.stderr:512+0 records out 2024-09-13T12:29:49.278 INFO:tasks.workunit.client.0.smithi032.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.264071 s, 2.0 GB/s 2024-09-13T12:29:49.278 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:255: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/wal2 --command bluefs-bdev-migrate 2024-09-13T12:29:49.297 INFO:tasks.workunit.client.0.smithi032.stdout:inferring bluefs devices from bluestore path 2024-09-13T12:29:55.377 INFO:tasks.workunit.client.0.smithi032.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-09-13T12:29:55.377 INFO:tasks.workunit.client.0.smithi032.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal2 2024-09-13T12:29:55.381 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:259: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/0/wal 2024-09-13T12:29:55.437 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:261: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-13T12:29:57.747 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:29:57.756 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:264: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-13T12:29:58.318 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:29:58.317+0000 7fe38640e980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:29:59.106 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:29:59.105+0000 7fe38640e980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:30:00.071 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:30:00.079 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:266: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db2 count=1024 bs=1M 2024-09-13T12:30:00.591 INFO:tasks.workunit.client.0.smithi032.stderr:1024+0 records in 2024-09-13T12:30:00.592 INFO:tasks.workunit.client.0.smithi032.stderr:1024+0 records out 2024-09-13T12:30:00.592 INFO:tasks.workunit.client.0.smithi032.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.510715 s, 2.1 GB/s 2024-09-13T12:30:00.592 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:267: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --devs-source td/osd-bluefs-volume-ops/1/block.wal --dev-target td/osd-bluefs-volume-ops/1/db2 --command bluefs-bdev-migrate 2024-09-13T12:30:00.611 INFO:tasks.workunit.client.0.smithi032.stdout:inferring bluefs devices from bluestore path 2024-09-13T12:30:01.668 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:01.667+0000 7f6c8a120980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:30:02.431 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:02.431+0000 7f6c8a120980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:30:08.852 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:08.851+0000 7f6c8a120980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:30:10.436 INFO:tasks.workunit.client.0.smithi032.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-09-13T12:30:10.436 INFO:tasks.workunit.client.0.smithi032.stdout: device removed:0 td/osd-bluefs-volume-ops/1/block.wal 2024-09-13T12:30:10.436 INFO:tasks.workunit.client.0.smithi032.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db2 2024-09-13T12:30:10.439 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:273: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db 2024-09-13T12:30:10.547 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:275: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-13T12:30:11.120 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:11.119+0000 7fd78b8f0980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:30:11.617 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:11.617+0000 7fd78b8f0980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:30:12.359 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:30:12.369 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:277: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-09-13T12:30:12.392 INFO:tasks.workunit.client.0.smithi032.stdout:inferring bluefs devices from bluestore path 2024-09-13T12:30:13.181 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:13.180+0000 7f5e96c11980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:30:13.693 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:13.692+0000 7f5e96c11980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:30:14.444 INFO:tasks.workunit.client.0.smithi032.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-09-13T12:30:14.449 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:282: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db2 2024-09-13T12:30:14.555 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:284: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-13T12:30:15.863 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:30:15.872 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:287: TEST_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-bluefs-volume-ops/2 --op fsck --no-mon-config 2024-09-13T12:30:17.934 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:30:17.943 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:290: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-09-13T12:30:18.457 INFO:tasks.workunit.client.0.smithi032.stderr:1024+0 records in 2024-09-13T12:30:18.457 INFO:tasks.workunit.client.0.smithi032.stderr:1024+0 records out 2024-09-13T12:30:18.457 INFO:tasks.workunit.client.0.smithi032.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.512667 s, 2.1 GB/s 2024-09-13T12:30:18.457 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:291: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-09-13T12:30:18.478 INFO:tasks.workunit.client.0.smithi032.stdout:inferring bluefs devices from bluestore path 2024-09-13T12:30:19.264 INFO:tasks.workunit.client.0.smithi032.stderr:Unable to migrate Slow volume to new location, please allocate new DB or WAL with --bluefs-bdev-new-db(wal) command 2024-09-13T12:30:19.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:297: TEST_bluestore: '[' 1 -eq 0 ']' 2024-09-13T12:30:19.266 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:300: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-13T12:30:19.819 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:19.819+0000 7f01f9e33980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:30:20.582 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:20.582+0000 7f01f9e33980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:30:21.579 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:30:21.588 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:303: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-09-13T12:30:28.642 INFO:tasks.workunit.client.0.smithi032.stderr:1024+0 records in 2024-09-13T12:30:28.642 INFO:tasks.workunit.client.0.smithi032.stderr:1024+0 records out 2024-09-13T12:30:28.642 INFO:tasks.workunit.client.0.smithi032.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.92867 s, 155 MB/s 2024-09-13T12:30:28.642 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:305: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --devs-source td/osd-bluefs-volume-ops/2/block.db --devs-source td/osd-bluefs-volume-ops/2/block.wal --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-09-13T12:30:28.665 INFO:tasks.workunit.client.0.smithi032.stdout:inferring bluefs devices from bluestore path 2024-09-13T12:30:29.717 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:29.715+0000 7f5e015cf980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:30:30.480 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:30.478+0000 7f5e015cf980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:30:30.520 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:30.518+0000 7f5e015cf980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:30:31.727 INFO:tasks.workunit.client.0.smithi032.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-09-13T12:30:31.728 INFO:tasks.workunit.client.0.smithi032.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-09-13T12:30:31.728 INFO:tasks.workunit.client.0.smithi032.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/2/db2 2024-09-13T12:30:31.732 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:312: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-13T12:30:32.283 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:32.281+0000 7fe18c91b980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:30:32.795 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:32.794+0000 7fe18c91b980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:30:33.543 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:30:33.551 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:315: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/3/wal2 count=1024 bs=1M 2024-09-13T12:30:34.059 INFO:tasks.workunit.client.0.smithi032.stderr:1024+0 records in 2024-09-13T12:30:34.059 INFO:tasks.workunit.client.0.smithi032.stderr:1024+0 records out 2024-09-13T12:30:34.059 INFO:tasks.workunit.client.0.smithi032.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.507024 s, 2.1 GB/s 2024-09-13T12:30:34.059 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:317: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/wal2 --command bluefs-bdev-migrate 2024-09-13T12:30:34.079 INFO:tasks.workunit.client.0.smithi032.stdout:inferring bluefs devices from bluestore path 2024-09-13T12:30:43.139 INFO:tasks.workunit.client.0.smithi032.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block 2024-09-13T12:30:43.139 INFO:tasks.workunit.client.0.smithi032.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-09-13T12:30:43.139 INFO:tasks.workunit.client.0.smithi032.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/3/wal2 2024-09-13T12:30:43.143 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:323: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-13T12:30:43.695 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:43.694+0000 7fa5c402d980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:30:44.260 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:44.259+0000 7fa5c402d980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:30:44.707 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:30:44.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:325: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-09-13T12:30:44.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:30:44.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:30:44.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T12:30:44.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:30:44.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-13T12:30:44.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-13T12:30:44.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:30:44.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:30:44.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:30:44.718 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-13T12:30:44.719 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-13T12:30:44.719 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:30:44.719 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:30:44.719 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-13T12:30:44.719 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:30:44.719 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:30:44.719 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:30:44.719 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:30:44.719 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:30:44.719 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:30:44.719 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:30:44.719 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:30:44.719 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:30:44.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-13T12:30:44.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-13T12:30:44.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:30:44.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:30:44.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:30:44.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:30:44.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:30:44.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:30:44.720 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-13T12:30:44.720 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.0 2024-09-13T12:30:44.721 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T12:30:44.721 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:30:44.721 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-13T12:30:44.722 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T12:30:44.722 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:30:44.722 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:30:44.722 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:30:44.746 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:44.745+0000 7f50ff41e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:30:44.748 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:44.747+0000 7f50ff41e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:30:44.749 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:44.749+0000 7f50ff41e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:30:45.149 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T12:30:45.149 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:30:45.150 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T12:30:45.150 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:30:45.150 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:30:45.150 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:30:45.150 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:30:45.151 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:30:45.151 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:30:45.151 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:30:45.594 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:30:46.595 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:30:46.595 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:30:46.596 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:30:46.596 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:30:46.596 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:30:46.596 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:30:46.626 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:46.625+0000 7f50ff41e640 -1 Falling back to public interface 2024-09-13T12:30:47.036 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:30:48.038 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:30:48.038 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:30:48.038 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:30:48.039 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:30:48.039 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:30:48.039 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:30:48.446 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:30:49.447 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:30:49.447 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:30:49.447 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:30:49.448 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:30:49.448 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:30:49.448 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:30:49.898 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:30:50.900 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:30:50.900 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:30:50.900 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:30:50.901 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:30:50.901 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:30:50.901 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:30:51.329 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:30:51.624 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:51.623+0000 7f50ff41e640 -1 osd.0 75 log_to_monitors true 2024-09-13T12:30:52.330 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:30:52.330 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:30:52.330 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T12:30:52.331 INFO:tasks.workunit.client.0.smithi032.stdout:5 2024-09-13T12:30:52.331 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:30:52.331 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:30:52.763 INFO:tasks.workunit.client.0.smithi032.stdout:osd.0 up in weight 1 up_from 86 up_thru 86 down_at 76 last_clean_interval [61,75) [v2:127.0.0.1:6802/2851557963,v1:127.0.0.1:6803/2851557963] [v2:127.0.0.1:6804/2851557963,v1:127.0.0.1:6805/2851557963] exists,up f0200d83-d430-4fd5-a4ea-2304279eaa4f 2024-09-13T12:30:52.764 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:30:52.764 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:30:52.764 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:30:52.764 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-13T12:30:52.768 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: osd_pid0=145924 2024-09-13T12:30:52.768 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:327: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-09-13T12:30:52.768 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:30:52.768 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:30:52.768 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T12:30:52.768 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:30:52.768 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-13T12:30:52.768 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-13T12:30:52.768 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:30:52.768 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:30:52.769 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:30:52.769 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-13T12:30:52.769 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-13T12:30:52.769 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:30:52.769 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:30:52.769 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-13T12:30:52.769 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:30:52.769 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:30:52.769 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:30:52.769 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:30:52.769 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:30:52.769 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:30:52.769 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:30:52.769 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:30:52.770 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:30:52.770 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-13T12:30:52.770 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-13T12:30:52.770 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:30:52.770 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:30:52.770 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:30:52.770 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:30:52.770 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:30:52.770 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:30:52.770 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-13T12:30:52.770 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.1 2024-09-13T12:30:52.771 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T12:30:52.771 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:30:52.771 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-09-13T12:30:52.772 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T12:30:52.772 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:30:52.772 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:30:52.772 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:30:52.794 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:52.793+0000 7f0f59dc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:30:52.796 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:52.795+0000 7f0f59dc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:30:52.797 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:52.796+0000 7f0f59dc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:30:53.173 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T12:30:53.173 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:30:53.174 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T12:30:53.174 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:30:53.174 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:30:53.174 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:30:53.174 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:30:53.174 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:30:53.175 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:30:53.175 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:30:53.617 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:30:54.384 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:54.382+0000 7f0f59dc5640 -1 Falling back to public interface 2024-09-13T12:30:54.618 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:30:54.618 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:30:54.619 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:30:54.619 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:30:54.619 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:30:54.619 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:30:55.060 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:30:56.061 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:30:56.062 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:30:56.062 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:30:56.062 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:30:56.063 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:30:56.063 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:30:56.504 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:30:57.447 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:57.446+0000 7f0f59dc5640 -1 osd.1 78 log_to_monitors true 2024-09-13T12:30:57.505 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:30:57.505 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:30:57.505 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:30:57.506 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:30:57.506 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:30:57.506 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T12:30:57.964 INFO:tasks.workunit.client.0.smithi032.stdout:osd.1 up in weight 1 up_from 89 up_thru 76 down_at 79 last_clean_interval [64,78) [v2:127.0.0.1:6810/357011204,v1:127.0.0.1:6811/357011204] [v2:127.0.0.1:6812/357011204,v1:127.0.0.1:6813/357011204] exists,up 3bf12ce9-d5e2-4321-92bc-ee79874612e2 2024-09-13T12:30:57.965 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:30:57.965 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:30:57.965 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:30:57.965 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-13T12:30:57.968 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: osd_pid1=146927 2024-09-13T12:30:57.968 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:329: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-09-13T12:30:57.968 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:30:57.968 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:30:57.968 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-13T12:30:57.969 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:30:57.969 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-13T12:30:57.969 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-13T12:30:57.969 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:30:57.969 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:30:57.969 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:30:57.969 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-13T12:30:57.969 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-13T12:30:57.969 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:30:57.969 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:30:57.969 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-13T12:30:57.969 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:30:57.969 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:30:57.969 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:30:57.970 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:30:57.970 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:30:57.970 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:30:57.970 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:30:57.970 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:30:57.970 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:30:57.970 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-13T12:30:57.970 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-13T12:30:57.970 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:30:57.970 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:30:57.970 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:30:57.970 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:30:57.970 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:30:57.970 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:30:57.971 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-13T12:30:57.971 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.2 2024-09-13T12:30:57.972 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-13T12:30:57.972 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:30:57.972 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-09-13T12:30:57.972 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-13T12:30:57.972 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:30:57.972 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:30:57.972 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:30:57.997 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:57.994+0000 7fd975814640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:30:58.121 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:58.120+0000 7fd975814640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:30:58.122 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:58.121+0000 7fd975814640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:30:58.447 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-13T12:30:58.448 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:30:58.448 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T12:30:58.448 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:30:58.448 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:30:58.448 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:30:58.448 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:30:58.449 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:30:58.449 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:30:58.449 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:30:58.935 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:30:59.486 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:30:59.485+0000 7fd975814640 -1 Falling back to public interface 2024-09-13T12:30:59.937 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:30:59.937 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:30:59.937 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:30:59.937 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:30:59.938 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:30:59.938 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:31:00.387 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:31:01.388 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:31:01.388 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:31:01.388 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:31:01.389 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:31:01.389 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:31:01.389 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:31:01.677 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:31:01.676+0000 7fd975814640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:31:01.831 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:31:02.191 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:31:02.190+0000 7fd975814640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:31:02.832 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:31:02.832 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:31:02.832 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:31:02.832 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:31:02.833 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:31:02.833 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:31:02.947 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:31:02.947+0000 7fd975814640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:31:03.256 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:31:03.466 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:31:03.465+0000 7fd975814640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:31:03.786 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:31:03.786+0000 7fd975814640 -1 osd.2 81 log_to_monitors true 2024-09-13T12:31:04.257 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:31:04.257 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:31:04.257 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:31:04.257 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:31:04.258 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:31:04.258 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T12:31:04.927 INFO:tasks.workunit.client.0.smithi032.stdout:osd.2 up in weight 1 up_from 93 up_thru 93 down_at 82 last_clean_interval [68,81) [v2:127.0.0.1:6818/1075183351,v1:127.0.0.1:6819/1075183351] [v2:127.0.0.1:6820/1075183351,v1:127.0.0.1:6821/1075183351] exists,up 78431f38-564b-40c5-9ca6-e92448355c2a 2024-09-13T12:31:04.927 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:31:04.927 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:31:04.927 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:31:04.928 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-13T12:31:04.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: osd_pid2=147887 2024-09-13T12:31:04.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:331: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-09-13T12:31:04.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:31:04.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:31:04.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-13T12:31:04.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:31:04.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-13T12:31:04.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-13T12:31:04.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:31:04.931 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:31:04.932 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:31:04.932 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-13T12:31:04.932 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-13T12:31:04.932 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:31:04.932 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:31:04.932 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-13T12:31:04.932 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:31:04.932 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:31:04.932 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:31:04.932 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:31:04.932 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:31:04.932 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:31:04.932 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:31:04.932 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:31:04.933 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:31:04.933 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-13T12:31:04.933 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-13T12:31:04.933 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:31:04.933 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:31:04.933 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:31:04.933 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:31:04.933 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:31:04.933 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:31:04.933 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-13T12:31:04.933 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.3 2024-09-13T12:31:04.934 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-13T12:31:04.934 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:31:04.934 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-09-13T12:31:04.934 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-13T12:31:04.935 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:31:04.935 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:31:04.935 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:31:04.958 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:31:04.957+0000 7f7788e17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:31:04.959 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:31:04.959+0000 7f7788e17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:31:04.961 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:31:04.960+0000 7f7788e17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:31:05.431 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-13T12:31:05.431 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:31:05.432 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T12:31:05.432 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:31:05.432 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:31:05.432 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:31:05.432 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:31:05.433 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:31:05.433 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:31:05.433 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:31:05.865 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:31:06.808 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:31:06.807+0000 7f7788e17640 -1 Falling back to public interface 2024-09-13T12:31:06.866 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:31:06.866 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:31:06.866 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:31:06.867 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:31:06.867 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:31:06.867 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:31:07.292 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:31:08.294 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:31:08.294 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:31:08.294 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:31:08.295 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:31:08.295 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:31:08.295 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:31:08.731 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:31:08.730+0000 7f7788e17640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:31:08.735 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:31:08.996 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:31:08.995+0000 7f7788e17640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:31:09.737 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:31:09.737 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:31:09.737 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:31:09.737 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:31:09.738 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:31:09.738 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:31:09.749 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:31:09.749+0000 7f7788e17640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:31:10.179 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:31:10.274 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:31:10.273+0000 7f7788e17640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-13T12:31:10.769 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:31:10.769+0000 7f7788e17640 -1 osd.3 83 log_to_monitors true 2024-09-13T12:31:11.180 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:31:11.180 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:31:11.180 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:31:11.180 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:31:11.181 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:31:11.181 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T12:31:11.817 INFO:tasks.workunit.client.0.smithi032.stdout:osd.3 up in weight 1 up_from 97 up_thru 97 down_at 84 last_clean_interval [73,83) [v2:127.0.0.1:6826/3478791843,v1:127.0.0.1:6827/3478791843] [v2:127.0.0.1:6828/3478791843,v1:127.0.0.1:6829/3478791843] exists,up fc41ef3f-07fe-4ce3-aecb-2e8f6a9edb19 2024-09-13T12:31:11.817 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:31:11.818 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:31:11.818 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:31:11.818 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-13T12:31:11.833 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: osd_pid3=148843 2024-09-13T12:31:11.833 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:335: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-13T12:31:11.865 INFO:tasks.workunit.client.0.smithi032.stdout:hints = 1 2024-09-13T12:31:11.865 INFO:tasks.workunit.client.0.smithi032.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-13T12:31:11.865 INFO:tasks.workunit.client.0.smithi032.stdout:Object prefix: benchmark_data_smithi032_149801 2024-09-13T12:31:11.865 INFO:tasks.workunit.client.0.smithi032.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T12:31:11.865 INFO:tasks.workunit.client.0.smithi032.stdout: 0 0 0 0 0 0 - 0 2024-09-13T12:31:12.865 INFO:tasks.workunit.client.0.smithi032.stdout: 1 16 66 50 0.195292 0.195312 0.0973608 0.303025 2024-09-13T12:31:13.865 INFO:tasks.workunit.client.0.smithi032.stdout: 2 16 179 163 0.318316 0.441406 0.0990188 0.188411 2024-09-13T12:31:14.865 INFO:tasks.workunit.client.0.smithi032.stdout: 3 16 302 286 0.372343 0.480469 0.100316 0.162613 2024-09-13T12:31:15.865 INFO:tasks.workunit.client.0.smithi032.stdout: 4 16 404 388 0.378849 0.398438 0.142347 0.160138 2024-09-13T12:31:16.865 INFO:tasks.workunit.client.0.smithi032.stdout: 5 16 527 511 0.399163 0.480469 0.0905311 0.155602 2024-09-13T12:31:17.866 INFO:tasks.workunit.client.0.smithi032.stdout: 6 16 639 623 0.405541 0.4375 0.150799 0.152978 2024-09-13T12:31:18.866 INFO:tasks.workunit.client.0.smithi032.stdout: 7 16 757 741 0.413445 0.460938 0.134015 0.149765 2024-09-13T12:31:19.866 INFO:tasks.workunit.client.0.smithi032.stdout: 8 16 851 835 0.407656 0.367188 0.385203 0.151214 2024-09-13T12:31:20.866 INFO:tasks.workunit.client.0.smithi032.stdout: 9 16 968 952 0.413135 0.457031 0.117724 0.150482 2024-09-13T12:31:21.866 INFO:tasks.workunit.client.0.smithi032.stdout: 10 16 1080 1064 0.415564 0.4375 0.115883 0.149785 2024-09-13T12:31:22.866 INFO:tasks.workunit.client.0.smithi032.stdout: 11 16 1196 1180 0.418972 0.453125 0.142345 0.148827 2024-09-13T12:31:23.867 INFO:tasks.workunit.client.0.smithi032.stdout: 12 16 1317 1301 0.423441 0.472656 0.132365 0.146938 2024-09-13T12:31:24.867 INFO:tasks.workunit.client.0.smithi032.stdout: 13 16 1433 1417 0.425721 0.453125 0.114746 0.145759 2024-09-13T12:31:25.867 INFO:tasks.workunit.client.0.smithi032.stdout: 14 16 1513 1497 0.417631 0.3125 0.130206 0.148811 2024-09-13T12:31:26.867 INFO:tasks.workunit.client.0.smithi032.stdout: 15 16 1642 1626 0.423378 0.503906 0.111561 0.147005 2024-09-13T12:31:27.867 INFO:tasks.workunit.client.0.smithi032.stdout: 16 16 1764 1748 0.4267 0.476562 0.125807 0.145807 2024-09-13T12:31:28.867 INFO:tasks.workunit.client.0.smithi032.stdout: 17 16 1896 1880 0.431926 0.515625 0.124864 0.14415 2024-09-13T12:31:29.867 INFO:tasks.workunit.client.0.smithi032.stdout: 18 16 2009 1993 0.43245 0.441406 0.125583 0.144192 2024-09-13T12:31:30.867 INFO:tasks.workunit.client.0.smithi032.stdout: 19 16 2122 2106 0.432918 0.441406 0.13489 0.14386 2024-09-13T12:31:31.868 INFO:tasks.workunit.client.0.smithi032.stdout:2024-09-13T12:31:31.867880+0000 min lat: 0.0704149 max lat: 0.89418 avg lat: 0.144375 2024-09-13T12:31:31.868 INFO:tasks.workunit.client.0.smithi032.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T12:31:31.868 INFO:tasks.workunit.client.0.smithi032.stdout: 20 16 2230 2214 0.432364 0.421875 0.145897 0.144375 2024-09-13T12:31:32.867 INFO:tasks.workunit.client.0.smithi032.stdout: 21 16 2347 2331 0.433537 0.457031 0.11065 0.143765 2024-09-13T12:31:33.868 INFO:tasks.workunit.client.0.smithi032.stdout: 22 16 2473 2457 0.4362 0.492188 0.142147 0.143105 2024-09-13T12:31:34.869 INFO:tasks.workunit.client.0.smithi032.stdout: 23 16 2577 2561 0.434896 0.40625 0.160927 0.142824 2024-09-13T12:31:35.868 INFO:tasks.workunit.client.0.smithi032.stdout: 24 16 2702 2686 0.437118 0.488281 0.100738 0.142651 2024-09-13T12:31:36.868 INFO:tasks.workunit.client.0.smithi032.stdout: 25 16 2786 2770 0.432756 0.328125 0.369121 0.142714 2024-09-13T12:31:37.868 INFO:tasks.workunit.client.0.smithi032.stdout: 26 16 2904 2888 0.433837 0.460938 0.116963 0.143688 2024-09-13T12:31:38.868 INFO:tasks.workunit.client.0.smithi032.stdout: 27 16 3031 3015 0.43614 0.496094 0.149435 0.143007 2024-09-13T12:31:39.868 INFO:tasks.workunit.client.0.smithi032.stdout: 28 16 3146 3130 0.436605 0.449219 0.119052 0.142771 2024-09-13T12:31:40.869 INFO:tasks.workunit.client.0.smithi032.stdout: 29 16 3259 3243 0.436769 0.441406 0.110876 0.142635 2024-09-13T12:31:41.869 INFO:tasks.workunit.client.0.smithi032.stdout: 30 15 3369 3354 0.436662 0.433594 0.149618 0.142814 2024-09-13T12:31:42.869 INFO:tasks.workunit.client.0.smithi032.stdout:Total time run: 30.0727 2024-09-13T12:31:42.869 INFO:tasks.workunit.client.0.smithi032.stdout:Total writes made: 3369 2024-09-13T12:31:42.869 INFO:tasks.workunit.client.0.smithi032.stdout:Write size: 4096 2024-09-13T12:31:42.869 INFO:tasks.workunit.client.0.smithi032.stdout:Object size: 4096 2024-09-13T12:31:42.870 INFO:tasks.workunit.client.0.smithi032.stdout:Bandwidth (MB/sec): 0.437612 2024-09-13T12:31:42.870 INFO:tasks.workunit.client.0.smithi032.stdout:Stddev Bandwidth: 0.0649798 2024-09-13T12:31:42.870 INFO:tasks.workunit.client.0.smithi032.stdout:Max bandwidth (MB/sec): 0.515625 2024-09-13T12:31:42.870 INFO:tasks.workunit.client.0.smithi032.stdout:Min bandwidth (MB/sec): 0.195312 2024-09-13T12:31:42.870 INFO:tasks.workunit.client.0.smithi032.stdout:Average IOPS: 112 2024-09-13T12:31:42.870 INFO:tasks.workunit.client.0.smithi032.stdout:Stddev IOPS: 16.6348 2024-09-13T12:31:42.870 INFO:tasks.workunit.client.0.smithi032.stdout:Max IOPS: 132 2024-09-13T12:31:42.870 INFO:tasks.workunit.client.0.smithi032.stdout:Min IOPS: 50 2024-09-13T12:31:42.870 INFO:tasks.workunit.client.0.smithi032.stdout:Average Latency(s): 0.142684 2024-09-13T12:31:42.870 INFO:tasks.workunit.client.0.smithi032.stdout:Stddev Latency(s): 0.0581141 2024-09-13T12:31:42.870 INFO:tasks.workunit.client.0.smithi032.stdout:Max latency(s): 0.89418 2024-09-13T12:31:42.870 INFO:tasks.workunit.client.0.smithi032.stdout:Min latency(s): 0.0704149 2024-09-13T12:31:42.922 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:337: TEST_bluestore: wait_for_clean 2024-09-13T12:31:42.922 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T12:31:42.922 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T12:31:42.922 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T12:31:42.923 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T12:31:42.923 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T12:31:42.923 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T12:31:42.924 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T12:31:42.924 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T12:31:42.924 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T12:31:43.024 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T12:31:43.024 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T12:31:43.024 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T12:31:43.024 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T12:31:43.024 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T12:31:43.024 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T12:31:43.456 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T12:31:43.456 INFO:tasks.workunit.client.0.smithi032.stderr:1 2024-09-13T12:31:43.456 INFO:tasks.workunit.client.0.smithi032.stderr:2 2024-09-13T12:31:43.456 INFO:tasks.workunit.client.0.smithi032.stderr:3' 2024-09-13T12:31:43.456 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T12:31:43.456 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:31:43.457 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T12:31:43.574 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=369367187469 2024-09-13T12:31:43.574 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 369367187469 2024-09-13T12:31:43.574 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-369367187469' 2024-09-13T12:31:43.574 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:31:43.575 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T12:31:43.689 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=382252089356 2024-09-13T12:31:43.689 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 382252089356 2024-09-13T12:31:43.689 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-369367187469 1-382252089356' 2024-09-13T12:31:43.689 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:31:43.690 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T12:31:43.805 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=399431958538 2024-09-13T12:31:43.805 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 399431958538 2024-09-13T12:31:43.805 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-369367187469 1-382252089356 2-399431958538' 2024-09-13T12:31:43.806 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:31:43.806 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T12:31:43.921 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=416611827721 2024-09-13T12:31:43.921 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 416611827721 2024-09-13T12:31:43.922 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-369367187469 1-382252089356 2-399431958538 3-416611827721' 2024-09-13T12:31:43.922 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:31:43.922 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-369367187469 2024-09-13T12:31:43.922 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:31:43.923 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T12:31:43.924 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-369367187469 2024-09-13T12:31:43.924 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:31:43.924 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=369367187469 2024-09-13T12:31:43.924 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 369367187469' 2024-09-13T12:31:43.925 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.0 seq 369367187469 2024-09-13T12:31:43.925 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:31:44.322 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 369367187469 -lt 369367187469 2024-09-13T12:31:44.322 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:31:44.322 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-382252089356 2024-09-13T12:31:44.322 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:31:44.323 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T12:31:44.324 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-382252089356 2024-09-13T12:31:44.324 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:31:44.325 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=382252089356 2024-09-13T12:31:44.325 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 382252089356' 2024-09-13T12:31:44.325 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.1 seq 382252089356 2024-09-13T12:31:44.325 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T12:31:44.758 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 382252089356 -lt 382252089356 2024-09-13T12:31:44.759 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:31:44.759 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-399431958538 2024-09-13T12:31:44.759 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:31:44.760 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T12:31:44.761 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-399431958538 2024-09-13T12:31:44.761 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:31:44.762 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=399431958538 2024-09-13T12:31:44.762 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.2 seq 399431958538 2024-09-13T12:31:44.762 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 399431958538' 2024-09-13T12:31:44.762 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T12:31:45.186 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 399431958539 -lt 399431958538 2024-09-13T12:31:45.186 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:31:45.186 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-416611827721 2024-09-13T12:31:45.186 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:31:45.187 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T12:31:45.187 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-416611827721 2024-09-13T12:31:45.188 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:31:45.188 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=416611827721 2024-09-13T12:31:45.189 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 416611827721' 2024-09-13T12:31:45.189 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.3 seq 416611827721 2024-09-13T12:31:45.189 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T12:31:45.619 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 416611827721 -lt 416611827721 2024-09-13T12:31:45.620 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T12:31:45.620 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:31:45.620 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:31:46.176 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 16 == 0 2024-09-13T12:31:46.176 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:31:46.177 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:31:46.177 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:31:46.177 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:31:46.177 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:31:46.177 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:31:46.177 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:31:46.577 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=16 2024-09-13T12:31:46.577 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:31:46.577 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:31:46.577 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:31:47.124 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 = 16 2024-09-13T12:31:47.124 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T12:31:47.124 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T12:31:47.124 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-09-13T12:31:47.125 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:31:47.125 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T12:31:47.125 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-13T12:31:47.125 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:31:47.125 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:31:47.125 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:31:47.125 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:31:47.125 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:31:47.233 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:31:47.233 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T12:31:47.234 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T12:31:47.234 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T12:31:47.236 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T12:31:47.236 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T12:31:47.236 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T12:31:47.237 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:31:47.237 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T12:31:47.237 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T12:31:47.237 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:31:47.238 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:31:47.239 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T12:31:47.239 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-13T12:31:47.684 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T12:31:47.684 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:31:47.684 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:31:47.684 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.102856 2024-09-13T12:31:47.686 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T12:31:47.686 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T12:31:47.686 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-09-13T12:31:47.686 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-09-13T12:31:47.686 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:31:47.686 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-bluefs-volume-ops 2024-09-13T12:31:47.686 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:31:47.686 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T12:31:47.686 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-13T12:31:47.686 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:31:47.686 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:31:47.686 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:31:47.687 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:31:47.687 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:31:47.687 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:31:47.688 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T12:31:47.688 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T12:31:47.689 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T12:31:47.690 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T12:31:47.690 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T12:31:47.690 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T12:31:47.691 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:31:47.691 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T12:31:47.691 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T12:31:47.691 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:31:47.692 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:31:47.693 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T12:31:47.693 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-13T12:31:47.694 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T12:31:47.695 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:31:47.695 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:31:47.695 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.102856 2024-09-13T12:31:47.695 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T12:31:47.695 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T12:31:47.695 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-bluefs-volume-ops 2024-09-13T12:31:47.697 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T12:31:47.697 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:31:47.697 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:31:47.697 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.102856 2024-09-13T12:31:47.699 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T12:31:47.699 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T12:31:47.699 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T12:31:47.699 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-09-13T12:31:47.699 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore2 td/osd-bluefs-volume-ops 2024-09-13T12:31:47.699 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:341: TEST_bluestore2: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:31:47.699 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: ulimit -n 2024-09-13T12:31:47.700 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: local flimit=4096 2024-09-13T12:31:47.700 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:344: TEST_bluestore2: '[' 4096 -lt 1536 ']' 2024-09-13T12:31:47.700 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: export CEPH_MON=127.0.0.1:7146 2024-09-13T12:31:47.700 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: CEPH_MON=127.0.0.1:7146 2024-09-13T12:31:47.700 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:348: TEST_bluestore2: export CEPH_ARGS 2024-09-13T12:31:47.700 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: uuidgen 2024-09-13T12:31:47.702 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: CEPH_ARGS+='--fsid=d4e415df-1430-4ac9-a0db-aa12422173c0 --auth-supported=none ' 2024-09-13T12:31:47.702 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:350: TEST_bluestore2: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-13T12:31:47.702 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:351: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-09-13T12:31:47.702 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:352: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-09-13T12:31:47.702 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:353: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-09-13T12:31:47.702 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:354: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-09-13T12:31:47.702 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:355: TEST_bluestore2: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-09-13T12:31:47.703 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:356: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-09-13T12:31:47.703 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:357: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-09-13T12:31:47.703 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:358: TEST_bluestore2: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-09-13T12:31:47.703 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:360: TEST_bluestore2: run_mon td/osd-bluefs-volume-ops a 2024-09-13T12:31:47.703 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:31:47.703 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T12:31:47.703 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T12:31:47.703 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T12:31:47.703 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-09-13T12:31:47.703 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-09-13T12:31:48.169 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T12:31:48.169 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:31:48.169 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:31:48.169 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:31:48.169 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:31:48.169 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:31:48.169 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:31:48.170 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-13T12:31:48.361 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T12:31:48.362 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T12:31:48.362 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T12:31:48.363 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T12:31:48.363 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T12:31:48.363 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T12:31:48.364 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T12:31:48.364 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T12:31:48.364 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T12:31:48.364 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:31:48.364 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:31:48.364 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:31:48.364 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.102856/ceph-mon.a.asok 2024-09-13T12:31:48.364 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T12:31:48.365 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.102856/ceph-mon.a.asok config get fsid 2024-09-13T12:31:48.449 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T12:31:48.450 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T12:31:48.450 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T12:31:48.450 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T12:31:48.450 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T12:31:48.450 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T12:31:48.450 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T12:31:48.450 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T12:31:48.451 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:31:48.451 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:31:48.451 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:31:48.451 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.102856/ceph-mon.a.asok 2024-09-13T12:31:48.451 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T12:31:48.451 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.102856/ceph-mon.a.asok config get mon_host 2024-09-13T12:31:48.531 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:361: TEST_bluestore2: run_mgr td/osd-bluefs-volume-ops x 2024-09-13T12:31:48.532 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:31:48.532 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T12:31:48.532 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T12:31:48.532 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T12:31:48.532 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-09-13T12:31:48.532 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T12:31:49.385 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T12:31:49.386 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:31:49.386 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:31:49.386 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:31:49.386 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:31:49.386 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:31:49.386 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:31:49.387 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T12:31:49.387 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T12:31:49.421 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:362: TEST_bluestore2: run_osd td/osd-bluefs-volume-ops 0 2024-09-13T12:31:49.421 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:31:49.421 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T12:31:49.422 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T12:31:49.422 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T12:31:49.422 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-13T12:31:49.422 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=d4e415df-1430-4ac9-a0db-aa12422173c0 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-13T12:31:49.422 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:31:49.422 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:31:49.422 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:31:49.422 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-13T12:31:49.422 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-13T12:31:49.422 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T12:31:49.422 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T12:31:49.422 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-13T12:31:49.423 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T12:31:49.423 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:31:49.423 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:31:49.423 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:31:49.423 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:31:49.423 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:31:49.423 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:31:49.423 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:31:49.423 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:31:49.423 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T12:31:49.423 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T12:31:49.423 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-13T12:31:49.423 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-13T12:31:49.423 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:31:49.424 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:31:49.424 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:31:49.424 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:31:49.424 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T12:31:49.424 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T12:31:49.424 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-13T12:31:49.424 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T12:31:49.424 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7a6ef068-a269-4c95-b03c-622605ce6b74 2024-09-13T12:31:49.425 INFO:tasks.workunit.client.0.smithi032.stdout:add osd0 7a6ef068-a269-4c95-b03c-622605ce6b74 2024-09-13T12:31:49.425 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 7a6ef068-a269-4c95-b03c-622605ce6b74' 2024-09-13T12:31:49.426 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T12:31:49.445 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC1MORmVyV2GhAADc6zbQDXGFvfVOEzFanCew== 2024-09-13T12:31:49.445 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC1MORmVyV2GhAADc6zbQDXGFvfVOEzFanCew=="}' 2024-09-13T12:31:49.445 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7a6ef068-a269-4c95-b03c-622605ce6b74 -i td/osd-bluefs-volume-ops/0/new.json 2024-09-13T12:31:49.704 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:31:49.714 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-09-13T12:31:49.715 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=d4e415df-1430-4ac9-a0db-aa12422173c0 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC1MORmVyV2GhAADc6zbQDXGFvfVOEzFanCew== --osd-uuid 7a6ef068-a269-4c95-b03c-622605ce6b74 2024-09-13T12:31:49.749 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:31:49.749+0000 7f840e61e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:31:49.763 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:31:49.762+0000 7f840e61e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:31:49.764 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:31:49.764+0000 7f840e61e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:31:49.765 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:31:49.764+0000 7f840e61e640 -1 bdev(0x558001a6c800 td/osd-bluefs-volume-ops/0/block) open stat got: (1) Operation not permitted 2024-09-13T12:31:49.765 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:31:49.764+0000 7f840e61e640 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-09-13T12:31:57.067 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-09-13T12:31:57.067 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T12:31:57.067 INFO:tasks.workunit.client.0.smithi032.stdout:adding osd0 key to auth repository 2024-09-13T12:31:57.068 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T12:31:57.068 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T12:31:57.648 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.0 2024-09-13T12:31:57.649 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T12:31:57.649 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=d4e415df-1430-4ac9-a0db-aa12422173c0 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:31:57.649 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T12:31:57.649 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T12:31:57.650 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T12:31:57.681 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:31:57.680+0000 7f40f4c85640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:31:57.684 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:31:57.683+0000 7f40f4c85640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:31:57.685 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:31:57.685+0000 7f40f4c85640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:31:58.107 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T12:31:58.107 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:31:58.107 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T12:31:58.107 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:31:58.107 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:31:58.108 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:31:58.108 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:31:58.108 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:31:58.108 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:31:58.108 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:31:58.565 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:31:59.537 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:31:59.535+0000 7f40f4c85640 -1 Falling back to public interface 2024-09-13T12:31:59.566 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:31:59.567 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:31:59.567 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:31:59.567 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:31:59.567 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:31:59.567 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:32:00.025 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:32:01.026 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:32:01.026 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:32:01.026 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:32:01.027 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:32:01.027 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:32:01.027 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:32:01.476 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:32:02.477 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:32:02.478 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:32:02.478 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:32:02.478 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:32:02.478 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:32:02.479 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:32:02.919 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:32:03.920 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:32:03.920 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:32:03.920 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:32:03.921 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:32:03.921 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:32:03.921 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:32:03.933 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:32:03.931+0000 7f40f4c85640 -1 osd.0 0 log_to_monitors true 2024-09-13T12:32:05.254 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:32:06.255 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:32:06.255 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:32:06.256 INFO:tasks.workunit.client.0.smithi032.stdout:5 2024-09-13T12:32:06.256 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T12:32:06.256 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:32:06.256 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:32:07.639 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:32:08.641 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:32:08.641 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:32:08.641 INFO:tasks.workunit.client.0.smithi032.stdout:6 2024-09-13T12:32:08.641 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T12:32:08.642 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:32:08.642 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:32:09.107 INFO:tasks.workunit.client.0.smithi032.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3846921356,v1:127.0.0.1:6803/3846921356] [v2:127.0.0.1:6804/3846921356,v1:127.0.0.1:6805/3846921356] exists,up 7a6ef068-a269-4c95-b03c-622605ce6b74 2024-09-13T12:32:09.108 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:32:09.108 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:32:09.108 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:32:09.108 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-13T12:32:09.109 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: osd_pid0=151677 2024-09-13T12:32:09.109 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:365: TEST_bluestore2: sleep 5 2024-09-13T12:32:14.110 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:366: TEST_bluestore2: create_pool foo 16 2024-09-13T12:32:14.110 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 16 2024-09-13T12:32:14.710 INFO:tasks.workunit.client.0.smithi032.stderr:pool 'foo' already exists 2024-09-13T12:32:14.722 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T12:32:15.724 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:368: TEST_bluestore2: retry = 0 2024-09-13T12:32:15.724 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 368: retry: command not found 2024-09-13T12:32:15.724 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:369: TEST_bluestore2: [[ '' -le 5 ]] 2024-09-13T12:32:15.724 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:371: TEST_bluestore2: timeout 60 rados bench -p foo 10 write --write-omap --no-cleanup 2024-09-13T12:32:15.769 INFO:tasks.workunit.client.0.smithi032.stdout:hints = 1 2024-09-13T12:32:15.770 INFO:tasks.workunit.client.0.smithi032.stdout:Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects 2024-09-13T12:32:15.771 INFO:tasks.workunit.client.0.smithi032.stdout:Object prefix: benchmark_data_smithi032_152969 2024-09-13T12:32:15.806 INFO:tasks.workunit.client.0.smithi032.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T12:32:15.806 INFO:tasks.workunit.client.0.smithi032.stdout: 0 0 0 0 0 0 - 0 2024-09-13T12:32:16.806 INFO:tasks.workunit.client.0.smithi032.stdout: 1 16 32 16 63.995 64 0.994497 0.942526 2024-09-13T12:32:17.807 INFO:tasks.workunit.client.0.smithi032.stdout: 2 16 49 33 65.9861 68 0.784979 0.853707 2024-09-13T12:32:18.807 INFO:tasks.workunit.client.0.smithi032.stdout: 3 16 65 49 65.3213 64 0.809728 0.844568 2024-09-13T12:32:19.807 INFO:tasks.workunit.client.0.smithi032.stdout: 4 16 81 65 64.9892 64 1.02859 0.877748 2024-09-13T12:32:20.807 INFO:tasks.workunit.client.0.smithi032.stdout: 5 16 97 81 64.7894 64 1.01915 0.889564 2024-09-13T12:32:21.807 INFO:tasks.workunit.client.0.smithi032.stdout: 6 16 113 97 64.6569 64 1.22794 0.957854 2024-09-13T12:32:22.807 INFO:tasks.workunit.client.0.smithi032.stdout: 7 16 129 113 64.5616 64 0.80213 0.94039 2024-09-13T12:32:23.807 INFO:tasks.workunit.client.0.smithi032.stdout: 8 16 144 128 63.9905 60 1.31424 0.981062 2024-09-13T12:32:24.807 INFO:tasks.workunit.client.0.smithi032.stdout: 9 16 161 145 64.4353 68 0.918786 0.966333 2024-09-13T12:32:25.808 INFO:tasks.workunit.client.0.smithi032.stdout: 10 16 161 145 57.9916 0 - 0.966333 2024-09-13T12:32:26.808 INFO:tasks.workunit.client.0.smithi032.stdout:Total time run: 10.0903 2024-09-13T12:32:26.808 INFO:tasks.workunit.client.0.smithi032.stdout:Total writes made: 161 2024-09-13T12:32:26.809 INFO:tasks.workunit.client.0.smithi032.stdout:Write size: 4194304 2024-09-13T12:32:26.809 INFO:tasks.workunit.client.0.smithi032.stdout:Object size: 4194304 2024-09-13T12:32:26.809 INFO:tasks.workunit.client.0.smithi032.stdout:Bandwidth (MB/sec): 63.8237 2024-09-13T12:32:26.809 INFO:tasks.workunit.client.0.smithi032.stdout:Stddev Bandwidth: 20.5047 2024-09-13T12:32:26.809 INFO:tasks.workunit.client.0.smithi032.stdout:Max bandwidth (MB/sec): 68 2024-09-13T12:32:26.810 INFO:tasks.workunit.client.0.smithi032.stdout:Min bandwidth (MB/sec): 0 2024-09-13T12:32:26.810 INFO:tasks.workunit.client.0.smithi032.stdout:Average IOPS: 15 2024-09-13T12:32:26.810 INFO:tasks.workunit.client.0.smithi032.stdout:Stddev IOPS: 5.12619 2024-09-13T12:32:26.810 INFO:tasks.workunit.client.0.smithi032.stdout:Max IOPS: 17 2024-09-13T12:32:26.810 INFO:tasks.workunit.client.0.smithi032.stdout:Min IOPS: 0 2024-09-13T12:32:26.810 INFO:tasks.workunit.client.0.smithi032.stdout:Average Latency(s): 0.996538 2024-09-13T12:32:26.811 INFO:tasks.workunit.client.0.smithi032.stdout:Stddev Latency(s): 0.208487 2024-09-13T12:32:26.811 INFO:tasks.workunit.client.0.smithi032.stdout:Max latency(s): 1.31424 2024-09-13T12:32:26.811 INFO:tasks.workunit.client.0.smithi032.stdout:Min latency(s): 0.225691 2024-09-13T12:32:26.853 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:374: TEST_bluestore2: sleep 10 2024-09-13T12:32:36.856 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-09-13T12:32:36.857 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: jq .bluefs.db_used_bytes 2024-09-13T12:32:36.996 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: db_used=1060110336 2024-09-13T12:32:36.996 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-09-13T12:32:36.997 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: jq .bluefs.slow_used_bytes 2024-09-13T12:32:37.126 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: spilled_over=92012544 2024-09-13T12:32:37.126 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:378: TEST_bluestore2: (( retry+=1 )) 2024-09-13T12:32:37.126 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: test 92012544 -eq 0 2024-09-13T12:32:37.126 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: break 2024-09-13T12:32:37.126 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:381: TEST_bluestore2: test 92012544 -gt 0 2024-09-13T12:32:37.126 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 151677 2024-09-13T12:32:37.126 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: sleep 1 2024-09-13T12:32:38.127 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 151677 2024-09-13T12:32:38.127 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 383: kill: (151677) - No such process 2024-09-13T12:32:38.127 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:384: TEST_bluestore2: ceph osd down 0 2024-09-13T12:32:38.570 INFO:tasks.workunit.client.0.smithi032.stderr:osd.0 is already down. 2024-09-13T12:32:38.582 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:386: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/block --command bluefs-bdev-migrate 2024-09-13T12:32:38.606 INFO:tasks.workunit.client.0.smithi032.stdout:inferring bluefs devices from bluestore path 2024-09-13T12:32:48.670 INFO:tasks.workunit.client.0.smithi032.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-09-13T12:32:48.688 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:391: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command bluefs-bdev-sizes 2024-09-13T12:32:48.716 INFO:tasks.workunit.client.0.smithi032.stdout:inferring bluefs devices from bluestore path 2024-09-13T12:32:50.492 INFO:tasks.workunit.client.0.smithi032.stdout:1 : device size 0xffffe000 : using 0x44cd8000(1.1 GiB) 2024-09-13T12:32:50.499 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:394: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command fsck 2024-09-13T12:32:53.060 INFO:tasks.workunit.client.0.smithi032.stdout:fsck success 2024-09-13T12:32:53.095 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:397: TEST_bluestore2: activate_osd td/osd-bluefs-volume-ops 0 2024-09-13T12:32:53.095 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:32:53.095 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T12:32:53.095 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T12:32:53.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T12:32:53.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-13T12:32:53.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=d4e415df-1430-4ac9-a0db-aa12422173c0 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-13T12:32:53.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:32:53.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:32:53.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:32:53.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-13T12:32:53.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-13T12:32:53.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T12:32:53.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T12:32:53.096 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-13T12:32:53.096 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T12:32:53.096 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:32:53.097 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:32:53.097 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:32:53.097 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:32:53.097 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:32:53.097 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:32:53.097 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:32:53.097 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:32:53.097 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-13T12:32:53.097 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-13T12:32:53.097 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:32:53.097 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:32:53.097 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:32:53.097 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:32:53.098 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T12:32:53.098 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T12:32:53.098 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-13T12:32:53.098 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.0 2024-09-13T12:32:53.099 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T12:32:53.099 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=d4e415df-1430-4ac9-a0db-aa12422173c0 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:32:53.099 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-13T12:32:53.099 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T12:32:53.099 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T12:32:53.099 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T12:32:53.099 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T12:32:53.128 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:32:53.126+0000 7f6cfbe10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:32:53.130 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:32:53.128+0000 7f6cfbe10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:32:53.131 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:32:53.129+0000 7f6cfbe10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:32:53.539 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T12:32:53.540 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:32:53.540 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T12:32:53.540 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:32:53.540 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:32:53.540 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:32:53.540 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:32:53.541 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:32:53.541 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:32:53.541 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:32:53.950 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:32:54.221 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:32:54.220+0000 7f6cfbe10640 -1 Falling back to public interface 2024-09-13T12:32:54.951 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:32:54.951 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:32:54.952 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:32:54.952 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:32:54.952 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:32:54.952 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:32:55.400 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:32:56.401 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:32:56.402 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:32:56.402 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:32:56.402 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:32:56.402 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:32:56.402 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:32:56.868 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:32:57.869 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:32:57.870 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:32:57.870 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:32:57.870 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:32:57.871 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:32:57.871 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:32:58.289 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:32:59.290 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:32:59.290 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:32:59.291 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:32:59.292 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:32:59.292 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:32:59.292 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:32:59.709 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:33:00.710 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:33:00.710 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:33:00.711 INFO:tasks.workunit.client.0.smithi032.stdout:5 2024-09-13T12:33:00.711 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T12:33:00.711 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:33:00.711 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:33:01.127 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:33:01.135 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:33:01.134+0000 7f6cfbe10640 -1 osd.0 10 log_to_monitors true 2024-09-13T12:33:02.129 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:33:02.129 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:33:02.129 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T12:33:02.129 INFO:tasks.workunit.client.0.smithi032.stdout:6 2024-09-13T12:33:02.129 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:33:02.130 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:33:02.562 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:33:03.564 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:33:03.564 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:33:03.564 INFO:tasks.workunit.client.0.smithi032.stdout:7 2024-09-13T12:33:03.565 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T12:33:03.565 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:33:03.565 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:33:04.389 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:33:05.391 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:33:05.391 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:33:05.391 INFO:tasks.workunit.client.0.smithi032.stdout:8 2024-09-13T12:33:05.392 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T12:33:05.392 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:33:05.392 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:33:05.812 INFO:tasks.workunit.client.0.smithi032.stdout:osd.0 up in weight 1 up_from 13 up_thru 13 down_at 11 last_clean_interval [5,10) [v2:127.0.0.1:6802/1296311663,v1:127.0.0.1:6803/1296311663] [v2:127.0.0.1:6804/1296311663,v1:127.0.0.1:6805/1296311663] exists,up 7a6ef068-a269-4c95-b03c-622605ce6b74 2024-09-13T12:33:05.812 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:33:05.813 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:33:05.813 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:33:05.813 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-13T12:33:05.814 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: osd_pid0=154200 2024-09-13T12:33:05.814 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:400: TEST_bluestore2: wait_for_clean 2024-09-13T12:33:05.814 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T12:33:05.814 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T12:33:05.814 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T12:33:05.814 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T12:33:05.815 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T12:33:05.815 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T12:33:05.815 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T12:33:05.815 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T12:33:05.815 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T12:33:05.906 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T12:33:05.907 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T12:33:05.907 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T12:33:05.907 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T12:33:05.907 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T12:33:05.907 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T12:33:06.358 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-13T12:33:06.358 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T12:33:06.358 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T12:33:06.358 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T12:33:06.480 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574852 2024-09-13T12:33:06.480 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574852 2024-09-13T12:33:06.481 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-55834574852' 2024-09-13T12:33:06.481 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T12:33:06.481 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-55834574852 2024-09-13T12:33:06.481 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T12:33:06.483 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T12:33:06.483 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-55834574852 2024-09-13T12:33:06.483 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T12:33:06.483 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574852 2024-09-13T12:33:06.483 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 55834574852' 2024-09-13T12:33:06.484 INFO:tasks.workunit.client.0.smithi032.stdout:waiting osd.0 seq 55834574852 2024-09-13T12:33:06.484 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:33:06.918 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 55834574852 2024-09-13T12:33:06.918 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T12:33:07.919 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T12:33:07.919 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T12:33:08.361 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574852 -lt 55834574852 2024-09-13T12:33:08.361 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T12:33:08.361 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:33:08.362 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:33:08.912 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 16 == 0 2024-09-13T12:33:08.912 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T12:33:08.912 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T12:33:08.912 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T12:33:08.912 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T12:33:08.912 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T12:33:08.912 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T12:33:08.913 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T12:33:09.328 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=16 2024-09-13T12:33:09.328 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T12:33:09.329 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T12:33:09.329 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T12:33:09.885 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 = 16 2024-09-13T12:33:09.885 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T12:33:09.886 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T12:33:09.886 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-09-13T12:33:09.886 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:33:09.886 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T12:33:09.886 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-13T12:33:09.886 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:33:09.886 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:33:09.886 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:33:09.886 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:33:09.886 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:33:09.991 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:33:09.991 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T12:33:09.992 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T12:33:09.992 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T12:33:09.993 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T12:33:09.993 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T12:33:09.994 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T12:33:09.994 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:33:09.995 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T12:33:09.995 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T12:33:09.995 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:33:09.996 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:33:09.997 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T12:33:09.997 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-13T12:33:10.071 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T12:33:10.071 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:33:10.071 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:33:10.071 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.102856 2024-09-13T12:33:10.073 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T12:33:10.073 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T12:33:10.073 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-09-13T12:33:10.073 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-09-13T12:33:10.073 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:33:10.073 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-bluefs-volume-ops 2024-09-13T12:33:10.073 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:33:10.073 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T12:33:10.073 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-13T12:33:10.073 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:33:10.073 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:33:10.074 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:33:10.074 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:33:10.074 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:33:10.074 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:33:10.075 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T12:33:10.075 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T12:33:10.076 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T12:33:10.077 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T12:33:10.077 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T12:33:10.077 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T12:33:10.078 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:33:10.078 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T12:33:10.078 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T12:33:10.078 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:33:10.079 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:33:10.080 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T12:33:10.080 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-13T12:33:10.081 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T12:33:10.082 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:33:10.082 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:33:10.082 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.102856 2024-09-13T12:33:10.082 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T12:33:10.082 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T12:33:10.083 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-bluefs-volume-ops 2024-09-13T12:33:10.084 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T12:33:10.084 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:33:10.084 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:33:10.084 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.102856 2024-09-13T12:33:10.086 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T12:33:10.086 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T12:33:10.086 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T12:33:10.086 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-09-13T12:33:10.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore_expand td/osd-bluefs-volume-ops 2024-09-13T12:33:10.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:404: TEST_bluestore_expand: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:33:10.087 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: ulimit -n 2024-09-13T12:33:10.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: local flimit=4096 2024-09-13T12:33:10.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:407: TEST_bluestore_expand: '[' 4096 -lt 1536 ']' 2024-09-13T12:33:10.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: export CEPH_MON=127.0.0.1:7146 2024-09-13T12:33:10.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: CEPH_MON=127.0.0.1:7146 2024-09-13T12:33:10.087 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:411: TEST_bluestore_expand: export CEPH_ARGS 2024-09-13T12:33:10.087 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: uuidgen 2024-09-13T12:33:10.089 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: CEPH_ARGS+='--fsid=013801b9-bb4e-4e0f-8a54-6ae88b05e592 --auth-supported=none ' 2024-09-13T12:33:10.089 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:413: TEST_bluestore_expand: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-13T12:33:10.089 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:414: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-09-13T12:33:10.089 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:415: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-09-13T12:33:10.089 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:416: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-09-13T12:33:10.089 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:417: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-09-13T12:33:10.089 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:418: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-09-13T12:33:10.089 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:419: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-09-13T12:33:10.089 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:420: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-09-13T12:33:10.089 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:421: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-09-13T12:33:10.089 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:423: TEST_bluestore_expand: run_mon td/osd-bluefs-volume-ops a 2024-09-13T12:33:10.089 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:33:10.090 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T12:33:10.090 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T12:33:10.090 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T12:33:10.090 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-09-13T12:33:10.090 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-09-13T12:33:10.594 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T12:33:10.594 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:33:10.594 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:33:10.594 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:33:10.594 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:33:10.594 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:33:10.594 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:33:10.595 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-13T12:33:10.794 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T12:33:10.796 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T12:33:10.796 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T12:33:10.796 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T12:33:10.796 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T12:33:10.796 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T12:33:10.796 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T12:33:10.796 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T12:33:10.796 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T12:33:10.797 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:33:10.797 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:33:10.797 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:33:10.797 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.102856/ceph-mon.a.asok 2024-09-13T12:33:10.797 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T12:33:10.798 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.102856/ceph-mon.a.asok config get fsid 2024-09-13T12:33:10.883 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T12:33:10.883 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T12:33:10.883 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T12:33:10.883 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T12:33:10.883 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T12:33:10.883 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T12:33:10.883 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T12:33:10.883 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T12:33:10.883 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T12:33:10.883 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:33:10.884 INFO:tasks.workunit.client.0.smithi032.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:33:10.884 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.102856/ceph-mon.a.asok 2024-09-13T12:33:10.884 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T12:33:10.884 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.102856/ceph-mon.a.asok config get mon_host 2024-09-13T12:33:10.964 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:424: TEST_bluestore_expand: run_mgr td/osd-bluefs-volume-ops x 2024-09-13T12:33:10.964 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:33:10.964 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T12:33:10.965 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T12:33:10.965 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T12:33:10.965 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-09-13T12:33:10.965 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T12:33:11.835 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T12:33:11.835 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:33:11.835 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:33:11.835 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:33:11.835 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:33:11.835 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:33:11.835 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:33:11.836 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T12:33:11.836 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T12:33:11.876 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:425: TEST_bluestore_expand: run_osd td/osd-bluefs-volume-ops 0 2024-09-13T12:33:11.876 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:33:11.877 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T12:33:11.877 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T12:33:11.877 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T12:33:11.877 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-13T12:33:11.877 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=d4e415df-1430-4ac9-a0db-aa12422173c0 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=013801b9-bb4e-4e0f-8a54-6ae88b05e592 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-13T12:33:11.877 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T12:33:11.877 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T12:33:11.877 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T12:33:11.877 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-13T12:33:11.877 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-13T12:33:11.877 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T12:33:11.878 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T12:33:11.878 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-13T12:33:11.878 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T12:33:11.878 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T12:33:11.878 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T12:33:11.878 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T12:33:11.878 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:33:11.878 INFO:tasks.workunit.client.0.smithi032.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:33:11.878 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:33:11.878 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' 2024-09-13T12:33:11.878 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T12:33:11.878 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T12:33:11.878 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T12:33:11.878 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-13T12:33:11.879 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-13T12:33:11.879 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T12:33:11.879 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T12:33:11.879 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T12:33:11.879 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T12:33:11.879 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T12:33:11.879 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T12:33:11.879 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-13T12:33:11.879 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T12:33:11.879 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0573456f-2298-46bf-8389-031889044f82 2024-09-13T12:33:11.879 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 0573456f-2298-46bf-8389-031889044f82' 2024-09-13T12:33:11.880 INFO:tasks.workunit.client.0.smithi032.stdout:add osd0 0573456f-2298-46bf-8389-031889044f82 2024-09-13T12:33:11.880 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T12:33:11.903 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAHMeRmTJvFNRAAX6ehAfpuIiLRMckJKrEcrg== 2024-09-13T12:33:11.903 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAHMeRmTJvFNRAAX6ehAfpuIiLRMckJKrEcrg=="}' 2024-09-13T12:33:11.903 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0573456f-2298-46bf-8389-031889044f82 -i td/osd-bluefs-volume-ops/0/new.json 2024-09-13T12:33:12.185 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:33:12.195 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-09-13T12:33:12.196 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=d4e415df-1430-4ac9-a0db-aa12422173c0 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=013801b9-bb4e-4e0f-8a54-6ae88b05e592 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAHMeRmTJvFNRAAX6ehAfpuIiLRMckJKrEcrg== --osd-uuid 0573456f-2298-46bf-8389-031889044f82 2024-09-13T12:33:12.237 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:33:12.236+0000 7f9504b20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:33:12.254 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:33:12.253+0000 7f9504b20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:33:12.255 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:33:12.255+0000 7f9504b20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:33:12.256 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:33:12.255+0000 7f9504b20640 -1 bdev(0x5587ddc74800 td/osd-bluefs-volume-ops/0/block) open stat got: (1) Operation not permitted 2024-09-13T12:33:12.256 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:33:12.255+0000 7f9504b20640 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-09-13T12:33:20.282 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-09-13T12:33:20.424 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T12:33:20.425 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T12:33:20.425 INFO:tasks.workunit.client.0.smithi032.stdout:adding osd0 key to auth repository 2024-09-13T12:33:20.426 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T12:33:20.873 INFO:tasks.workunit.client.0.smithi032.stdout:start osd.0 2024-09-13T12:33:20.874 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T12:33:20.874 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=a144f380-3ad4-4389-95cb-e2fdaadb8673 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=d4e415df-1430-4ac9-a0db-aa12422173c0 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=013801b9-bb4e-4e0f-8a54-6ae88b05e592 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102856/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T12:33:20.874 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T12:33:20.874 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T12:33:20.874 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T12:33:20.912 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:33:20.911+0000 7f7eb60b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:33:20.913 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:33:20.913+0000 7f7eb60b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:33:20.915 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:33:20.914+0000 7f7eb60b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T12:33:21.329 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T12:33:21.329 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T12:33:21.329 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T12:33:21.329 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T12:33:21.329 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T12:33:21.329 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:33:21.330 INFO:tasks.workunit.client.0.smithi032.stdout:0 2024-09-13T12:33:21.330 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T12:33:21.330 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:33:21.330 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:33:21.771 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:33:22.758 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:33:22.757+0000 7f7eb60b0640 -1 Falling back to public interface 2024-09-13T12:33:22.772 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:33:22.773 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:33:22.773 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T12:33:22.773 INFO:tasks.workunit.client.0.smithi032.stdout:1 2024-09-13T12:33:22.773 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:33:22.773 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:33:23.230 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:33:24.231 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:33:24.231 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:33:24.232 INFO:tasks.workunit.client.0.smithi032.stdout:2 2024-09-13T12:33:24.232 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T12:33:24.232 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:33:24.232 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:33:24.681 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:33:25.682 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:33:25.683 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:33:25.683 INFO:tasks.workunit.client.0.smithi032.stdout:3 2024-09-13T12:33:25.683 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T12:33:25.683 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:33:25.684 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:33:26.133 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:33:27.134 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:33:27.134 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:33:27.135 INFO:tasks.workunit.client.0.smithi032.stdout:4 2024-09-13T12:33:27.135 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T12:33:27.135 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:33:27.135 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:33:27.168 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:33:27.168+0000 7f7eb60b0640 -1 osd.0 0 log_to_monitors true 2024-09-13T12:33:28.437 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:33:29.438 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:33:29.438 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:33:29.438 INFO:tasks.workunit.client.0.smithi032.stdout:5 2024-09-13T12:33:29.439 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T12:33:29.439 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:33:29.439 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:33:30.525 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T12:33:31.526 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T12:33:31.526 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T12:33:31.527 INFO:tasks.workunit.client.0.smithi032.stdout:6 2024-09-13T12:33:31.527 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T12:33:31.527 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T12:33:31.527 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T12:33:32.031 INFO:tasks.workunit.client.0.smithi032.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/446335980,v1:127.0.0.1:6803/446335980] [v2:127.0.0.1:6804/446335980,v1:127.0.0.1:6805/446335980] exists,up 0573456f-2298-46bf-8389-031889044f82 2024-09-13T12:33:32.032 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T12:33:32.032 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T12:33:32.032 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T12:33:32.032 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-13T12:33:32.032 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: osd_pid0=156959 2024-09-13T12:33:32.032 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:428: TEST_bluestore_expand: sleep 5 2024-09-13T12:33:37.034 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:429: TEST_bluestore_expand: create_pool foo 16 2024-09-13T12:33:37.034 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 16 2024-09-13T12:33:37.645 INFO:tasks.workunit.client.0.smithi032.stderr:pool 'foo' already exists 2024-09-13T12:33:37.657 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T12:33:38.659 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:432: TEST_bluestore_expand: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-13T12:33:38.706 INFO:tasks.workunit.client.0.smithi032.stdout:hints = 1 2024-09-13T12:33:38.706 INFO:tasks.workunit.client.0.smithi032.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-13T12:33:38.706 INFO:tasks.workunit.client.0.smithi032.stdout:Object prefix: benchmark_data_smithi032_157995 2024-09-13T12:33:38.706 INFO:tasks.workunit.client.0.smithi032.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T12:33:38.706 INFO:tasks.workunit.client.0.smithi032.stdout: 0 0 0 0 0 0 - 0 2024-09-13T12:33:39.706 INFO:tasks.workunit.client.0.smithi032.stdout: 1 16 128 112 0.437445 0.4375 0.126043 0.138198 2024-09-13T12:33:40.706 INFO:tasks.workunit.client.0.smithi032.stdout: 2 16 245 229 0.4472 0.457031 0.117045 0.132353 2024-09-13T12:33:41.706 INFO:tasks.workunit.client.0.smithi032.stdout: 3 16 373 357 0.464774 0.5 0.117324 0.13108 2024-09-13T12:33:42.706 INFO:tasks.workunit.client.0.smithi032.stdout: 4 16 501 485 0.473568 0.5 0.117242 0.129086 2024-09-13T12:33:43.707 INFO:tasks.workunit.client.0.smithi032.stdout: 5 16 613 597 0.466344 0.4375 0.125532 0.131358 2024-09-13T12:33:44.707 INFO:tasks.workunit.client.0.smithi032.stdout: 6 16 741 725 0.471939 0.5 0.117144 0.13121 2024-09-13T12:33:45.707 INFO:tasks.workunit.client.0.smithi032.stdout: 7 16 837 821 0.458082 0.375 0.11745 0.131107 2024-09-13T12:33:46.707 INFO:tasks.workunit.client.0.smithi032.stdout: 8 16 960 944 0.460871 0.480469 0.117226 0.134677 2024-09-13T12:33:47.707 INFO:tasks.workunit.client.0.smithi032.stdout: 9 16 1077 1061 0.460436 0.457031 0.13408 0.134273 2024-09-13T12:33:48.707 INFO:tasks.workunit.client.0.smithi032.stdout: 10 16 1205 1189 0.464385 0.5 0.150734 0.133716 2024-09-13T12:33:49.707 INFO:tasks.workunit.client.0.smithi032.stdout: 11 16 1328 1312 0.465842 0.480469 0.126104 0.133744 2024-09-13T12:33:50.707 INFO:tasks.workunit.client.0.smithi032.stdout: 12 16 1445 1429 0.465104 0.457031 0.133725 0.133484 2024-09-13T12:33:51.708 INFO:tasks.workunit.client.0.smithi032.stdout: 13 16 1557 1541 0.462976 0.4375 0.125694 0.134276 2024-09-13T12:33:52.708 INFO:tasks.workunit.client.0.smithi032.stdout: 14 16 1685 1669 0.465615 0.5 0.117276 0.133611 2024-09-13T12:33:53.708 INFO:tasks.workunit.client.0.smithi032.stdout: 15 16 1808 1792 0.4666 0.480469 0.125633 0.133441 2024-09-13T12:33:54.708 INFO:tasks.workunit.client.0.smithi032.stdout: 16 16 1925 1909 0.465998 0.457031 0.108761 0.133219 2024-09-13T12:33:55.708 INFO:tasks.workunit.client.0.smithi032.stdout: 17 16 2048 2032 0.466846 0.480469 0.117583 0.133703 2024-09-13T12:33:56.708 INFO:tasks.workunit.client.0.smithi032.stdout: 18 16 2160 2144 0.465211 0.4375 0.125845 0.133904 2024-09-13T12:33:57.709 INFO:tasks.workunit.client.0.smithi032.stdout: 19 16 2272 2256 0.463749 0.4375 0.125862 0.134264 2024-09-13T12:33:58.709 INFO:tasks.workunit.client.0.smithi032.stdout:2024-09-13T12:33:58.709170+0000 min lat: 0.0667496 max lat: 0.384128 avg lat: 0.133986 2024-09-13T12:33:58.709 INFO:tasks.workunit.client.0.smithi032.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T12:33:58.709 INFO:tasks.workunit.client.0.smithi032.stdout: 20 16 2400 2384 0.465558 0.5 0.116923 0.133986 2024-09-13T12:33:59.709 INFO:tasks.workunit.client.0.smithi032.stdout: 21 16 2528 2512 0.467194 0.5 0.117462 0.133563 2024-09-13T12:34:00.709 INFO:tasks.workunit.client.0.smithi032.stdout: 22 16 2645 2629 0.466729 0.457031 0.125535 0.133651 2024-09-13T12:34:01.709 INFO:tasks.workunit.client.0.smithi032.stdout: 23 16 2757 2741 0.465456 0.4375 0.117464 0.133945 2024-09-13T12:34:02.709 INFO:tasks.workunit.client.0.smithi032.stdout: 24 16 2885 2869 0.466892 0.5 0.108904 0.13362 2024-09-13T12:34:03.709 INFO:tasks.workunit.client.0.smithi032.stdout: 25 16 3008 2992 0.467434 0.480469 0.134181 0.13353 2024-09-13T12:34:04.710 INFO:tasks.workunit.client.0.smithi032.stdout: 26 16 3120 3104 0.46628 0.4375 0.125915 0.133674 2024-09-13T12:34:05.710 INFO:tasks.workunit.client.0.smithi032.stdout: 27 16 3237 3221 0.465936 0.457031 0.125744 0.133717 2024-09-13T12:34:06.710 INFO:tasks.workunit.client.0.smithi032.stdout: 28 16 3376 3360 0.468684 0.542969 0.100569 0.133196 2024-09-13T12:34:07.710 INFO:tasks.workunit.client.0.smithi032.stdout: 29 16 3493 3477 0.46828 0.457031 0.117148 0.133238 2024-09-13T12:34:08.710 INFO:tasks.workunit.client.0.smithi032.stdout: 30 16 3616 3600 0.468684 0.480469 0.108977 0.133175 2024-09-13T12:34:09.710 INFO:tasks.workunit.client.0.smithi032.stdout:Total time run: 30.0841 2024-09-13T12:34:09.711 INFO:tasks.workunit.client.0.smithi032.stdout:Total writes made: 3616 2024-09-13T12:34:09.711 INFO:tasks.workunit.client.0.smithi032.stdout:Write size: 4096 2024-09-13T12:34:09.711 INFO:tasks.workunit.client.0.smithi032.stdout:Object size: 4096 2024-09-13T12:34:09.711 INFO:tasks.workunit.client.0.smithi032.stdout:Bandwidth (MB/sec): 0.469517 2024-09-13T12:34:09.711 INFO:tasks.workunit.client.0.smithi032.stdout:Stddev Bandwidth: 0.032553 2024-09-13T12:34:09.711 INFO:tasks.workunit.client.0.smithi032.stdout:Max bandwidth (MB/sec): 0.542969 2024-09-13T12:34:09.711 INFO:tasks.workunit.client.0.smithi032.stdout:Min bandwidth (MB/sec): 0.375 2024-09-13T12:34:09.711 INFO:tasks.workunit.client.0.smithi032.stdout:Average IOPS: 120 2024-09-13T12:34:09.711 INFO:tasks.workunit.client.0.smithi032.stdout:Stddev IOPS: 8.33356 2024-09-13T12:34:09.711 INFO:tasks.workunit.client.0.smithi032.stdout:Max IOPS: 139 2024-09-13T12:34:09.711 INFO:tasks.workunit.client.0.smithi032.stdout:Min IOPS: 96 2024-09-13T12:34:09.711 INFO:tasks.workunit.client.0.smithi032.stdout:Average Latency(s): 0.133043 2024-09-13T12:34:09.711 INFO:tasks.workunit.client.0.smithi032.stdout:Stddev Latency(s): 0.0275061 2024-09-13T12:34:09.712 INFO:tasks.workunit.client.0.smithi032.stdout:Max latency(s): 0.384128 2024-09-13T12:34:09.712 INFO:tasks.workunit.client.0.smithi032.stdout:Min latency(s): 0.0667496 2024-09-13T12:34:09.768 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:433: TEST_bluestore_expand: sleep 5 2024-09-13T12:34:14.771 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: ceph tell osd.0 perf dump bluefs 2024-09-13T12:34:14.772 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: jq .bluefs.slow_total_bytes 2024-09-13T12:34:14.917 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: total_space_before=4294967296 2024-09-13T12:34:14.918 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: ceph tell osd.0 bluestore bluefs device info 2024-09-13T12:34:14.918 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep BDEV_SLOW -A 2 2024-09-13T12:34:14.918 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep free 2024-09-13T12:34:14.918 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d: -f 2 2024-09-13T12:34:14.919 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d, -f 1 2024-09-13T12:34:14.919 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut '-d ' -f 2 2024-09-13T12:34:15.061 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: free_space_before=4280049664 2024-09-13T12:34:15.061 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 156959 2024-09-13T12:34:15.061 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: sleep 1 2024-09-13T12:34:16.062 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 156959 2024-09-13T12:34:16.063 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 439: kill: (156959) - No such process 2024-09-13T12:34:16.063 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:440: TEST_bluestore_expand: ceph osd down 0 2024-09-13T12:34:16.513 INFO:tasks.workunit.client.0.smithi032.stderr:osd.0 is already down. 2024-09-13T12:34:16.525 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:443: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 allocmap 2024-09-13T12:34:16.601 INFO:tasks.workunit.client.0.smithi032.stdout:allocmap bluestore.allocmap 2024-09-13T12:34:18.904 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:34:18.903+0000 7f612e9ee980 -1 bluestore::NCB::operator()::(2)compare_allocators:: spillover 2024-09-13T12:34:18.904 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:34:18.903+0000 7f612e9ee980 -1 bluestore::NCB::compare_allocators::mismatch:: idx1=7 idx2=8 2024-09-13T12:34:18.904 INFO:tasks.workunit.client.0.smithi032.stderr:2024-09-13T12:34:18.903+0000 7f612e9ee980 -1 bluestore::NCB::read_allocation_from_drive_for_bluestore_tool::FAILURE. Allocator from file and allocator from metadata differ::ret=-1 2024-09-13T12:34:19.346 INFO:tasks.workunit.client.0.smithi032.stderr:allocmap failed: (1) Operation not permitted 2024-09-13T12:34:19.352 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:443: TEST_bluestore_expand: return 1 2024-09-13T12:34:19.352 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: return 1 2024-09-13T12:34:19.352 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: code=1 2024-09-13T12:34:19.352 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-bluefs-volume-ops 1 2024-09-13T12:34:19.352 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-13T12:34:19.352 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=1 2024-09-13T12:34:19.353 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-13T12:34:19.353 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T12:34:19.353 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T12:34:19.353 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T12:34:19.353 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T12:34:19.353 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T12:34:19.459 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T12:34:19.459 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T12:34:19.460 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T12:34:19.460 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T12:34:19.461 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T12:34:19.461 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T12:34:19.461 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T12:34:19.462 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:34:19.462 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T12:34:19.463 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T12:34:19.463 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:34:19.464 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:34:19.465 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 1 = 1 ']' 2024-09-13T12:34:19.465 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:181: teardown: '[' -n '' ']' 2024-09-13T12:34:19.465 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:185: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-09-13T12:34:19.466 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:186: teardown: mv td/osd-bluefs-volume-ops/bluestore_tool.log td/osd-bluefs-volume-ops/mgr.x.log td/osd-bluefs-volume-ops/mon.a.log td/osd-bluefs-volume-ops/osd.0.log /home/ubuntu/cephtest/archive/log 2024-09-13T12:34:19.468 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-13T12:34:19.470 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T12:34:19.471 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T12:34:19.471 INFO:tasks.workunit.client.0.smithi032.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102856 2024-09-13T12:34:19.471 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.102856 2024-09-13T12:34:19.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T12:34:19.472 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T12:34:19.472 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T12:34:19.473 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 1 2024-09-13T12:34:19.473 INFO:tasks.workunit:Stopping ['osd'] on client.0... 2024-09-13T12:34:19.473 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-13T12:34:19.819 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on smithi032 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d58f73069a8157a9ace12a0247ee3e8497707ea9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh' 2024-09-13T12:34:20.171 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=b574c4546a7148e78f80412d69881d98 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on smithi032 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d58f73069a8157a9ace12a0247ee3e8497707ea9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh' 2024-09-13T12:34:20.174 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-13T12:34:20.186 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-13T12:34:20.186 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T12:34:20.262 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-13T12:34:20.271 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-13T12:34:20.271 DEBUG:teuthology.orchestra.run.smithi032:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T12:34:20.287 INFO:teuthology.orchestra.run.smithi032.stderr:bash: line 1: ntpq: command not found 2024-09-13T12:34:20.291 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T12:34:20.291 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2024-09-13T12:34:20.291 INFO:teuthology.orchestra.run.smithi032.stdout:^* hv01.front.sepia.ceph.com 3 6 377 65 -93us[ -126us] +/- 44ms 2024-09-13T12:34:20.291 INFO:teuthology.orchestra.run.smithi032.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 2 +483us[ +483us] +/- 64ms 2024-09-13T12:34:20.292 INFO:teuthology.orchestra.run.smithi032.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 -370us[ -370us] +/- 63ms 2024-09-13T12:34:20.292 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T12:34:20.293 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-13T12:34:20.301 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-13T12:34:20.302 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-13T12:34:20.309 DEBUG:teuthology.orchestra.run.smithi032:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T12:34:20.394 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T12:34:20.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T12:34:20.422 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-13T12:34:20.431 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-13T12:34:20.437 INFO:teuthology.task.internal:Duration was 2694.351149 seconds 2024-09-13T12:34:20.438 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-13T12:34:20.446 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-13T12:34:20.446 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T12:34:20.502 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T12:34:20.928 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-13T12:34:20.928 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi032.front.sepia.ceph.com 2024-09-13T12:34:20.928 DEBUG:teuthology.orchestra.run.smithi032:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T12:34:20.991 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-13T12:34:20.992 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T12:34:21.060 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-13T12:34:21.060 DEBUG:teuthology.orchestra.run.smithi032:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T12:34:21.201 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-13T12:34:21.211 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-13T12:34:21.211 DEBUG:teuthology.orchestra.run.smithi032:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T12:34:21.236 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-13T12:34:21.245 DEBUG:teuthology.orchestra.run.smithi032:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:34:21.300 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = core 2024-09-13T12:34:21.321 DEBUG:teuthology.orchestra.run.smithi032:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:34:21.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T12:34:21.377 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-13T12:34:21.385 INFO:teuthology.task.internal:Transferring archived files... 2024-09-13T12:34:21.387 DEBUG:teuthology.misc:Transferring archived files from smithi032:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903460/remote/smithi032 2024-09-13T12:34:21.388 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T12:34:24.578 INFO:teuthology.task.internal:Removing archive directory... 2024-09-13T12:34:24.579 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T12:34:24.646 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-13T12:34:24.657 INFO:teuthology.task.internal:Not uploading archives. 2024-09-13T12:34:24.658 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-13T12:34:24.676 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-13T12:34:24.677 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T12:34:24.704 INFO:teuthology.orchestra.run.smithi032.stdout: 265340 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 13 12:34 /home/ubuntu/cephtest 2024-09-13T12:34:24.704 INFO:teuthology.orchestra.run.smithi032.stdout: 280023 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 13 11:55 /home/ubuntu/cephtest/mnt.0 2024-09-13T12:34:24.705 INFO:teuthology.orchestra.run.smithi032.stdout: 280024 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 13 12:21 /home/ubuntu/cephtest/mnt.0/client.0 2024-09-13T12:34:24.705 INFO:teuthology.orchestra.run.smithi032.stdout: 393235 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 13 12:21 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-13T12:34:24.705 INFO:teuthology.orchestra.run.smithi032.stdout: 393236 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 12:34 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-09-13T12:34:24.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T12:34:24.706 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-13T12:34:24.707 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-13T12:34:24.712 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-13T12:34:24.721 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-13T12:34:24.756 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi032 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-13T12:34:24.760 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd} duration: 2694.3511488437653 failure_reason: 'Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on smithi032 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d58f73069a8157a9ace12a0247ee3e8497707ea9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh''' flavor: default owner: scheduled_akupczyk@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=b574c4546a7148e78f80412d69881d98 status: fail success: false 2024-09-13T12:34:24.760 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T12:34:24.862 INFO:teuthology.run:FAIL