2024-09-16T16:31:24.573 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T16:31:24.590 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T16:31:24.678 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2024-09-16_16:23:48-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi/7907376 branch: wip-matanb-crimson-testing-59543 description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_lock_and_fence} email: null first_in_suite: false job_id: '7907376' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2024-09-16_16:23:48-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-matanb-crimson-testing-59543 ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mgr stats period: 30 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd pool default crimson: true osd: crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug ms: 1 debug osd: 20 osd objectstore: seastore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-1 osd.1: crimson seastar cpu cores: 2-3 osd.2: crimson seastar cpu cores: 4-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: 81fedcb3b484c843d22ffbc8d179cb567b7083b5 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: crimson sha1: 81fedcb3b484c843d22ffbc8d179cb567b7083b5 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 0325fd3c11a225c0fd9eb38fab88aad1370ad779 owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 6678 sha1: 81fedcb3b484c843d22ffbc8d179cb567b7083b5 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 0325fd3c11a225c0fd9eb38fab88aad1370ad779 targets: smithi083.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOd2VVGJeFxu1bFV1Fg98tvaRHY33+WNcqLgarxVlS+AjgSryXYKMVYMrY7vKKFz3hN/KIp6o2da0Rylkz9tLfk= tasks: - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 flavor: crimson - workunit: clients: client.0: - rbd/test_lock_fence.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-16_16:23:48 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T16:31:24.678 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa; will attempt to use it 2024-09-16T16:31:24.679 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks 2024-09-16T16:31:24.679 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T16:31:24.681 INFO:teuthology.task.internal:Checking packages... 2024-09-16T16:31:24.703 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '81fedcb3b484c843d22ffbc8d179cb567b7083b5' 2024-09-16T16:31:24.703 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T16:31:24.703 INFO:teuthology.packaging:ref: None 2024-09-16T16:31:24.704 INFO:teuthology.packaging:tag: None 2024-09-16T16:31:24.704 INFO:teuthology.packaging:branch: wip-matanb-crimson-testing-59543 2024-09-16T16:31:24.704 INFO:teuthology.packaging:sha1: 81fedcb3b484c843d22ffbc8d179cb567b7083b5 2024-09-16T16:31:24.704 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-matanb-crimson-testing-59543 2024-09-16T16:31:24.844 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4879.g81fedcb3 2024-09-16T16:31:24.846 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T16:31:24.855 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T16:31:24.855 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T16:31:24.864 INFO:teuthology.task.internal:Saving configuration 2024-09-16T16:31:24.875 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T16:31:24.884 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T16:31:24.905 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-09-16_16:23:48-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi/7907376', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 16:24:53.115487', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOd2VVGJeFxu1bFV1Fg98tvaRHY33+WNcqLgarxVlS+AjgSryXYKMVYMrY7vKKFz3hN/KIp6o2da0Rylkz9tLfk='} 2024-09-16T16:31:24.908 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T16:31:24.916 INFO:teuthology.task.internal:roles: ubuntu@smithi083.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-16T16:31:24.916 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T16:31:24.951 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f91f4fc7be0>, signals=[15]) 2024-09-16T16:31:24.951 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T16:31:24.959 INFO:teuthology.task.internal:Opening connections... 2024-09-16T16:31:24.959 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi083.front.sepia.ceph.com 2024-09-16T16:31:24.960 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T16:31:25.035 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T16:31:25.041 DEBUG:teuthology.orchestra.run.smithi083:> uname -m 2024-09-16T16:31:25.085 INFO:teuthology.orchestra.run.smithi083.stdout:x86_64 2024-09-16T16:31:25.085 DEBUG:teuthology.orchestra.run.smithi083:> cat /etc/os-release 2024-09-16T16:31:25.142 INFO:teuthology.orchestra.run.smithi083.stdout:NAME="CentOS Stream" 2024-09-16T16:31:25.142 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION="9" 2024-09-16T16:31:25.142 INFO:teuthology.orchestra.run.smithi083.stdout:ID="centos" 2024-09-16T16:31:25.142 INFO:teuthology.orchestra.run.smithi083.stdout:ID_LIKE="rhel fedora" 2024-09-16T16:31:25.142 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION_ID="9" 2024-09-16T16:31:25.143 INFO:teuthology.orchestra.run.smithi083.stdout:PLATFORM_ID="platform:el9" 2024-09-16T16:31:25.143 INFO:teuthology.orchestra.run.smithi083.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T16:31:25.143 INFO:teuthology.orchestra.run.smithi083.stdout:ANSI_COLOR="0;31" 2024-09-16T16:31:25.143 INFO:teuthology.orchestra.run.smithi083.stdout:LOGO="fedora-logo-icon" 2024-09-16T16:31:25.143 INFO:teuthology.orchestra.run.smithi083.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T16:31:25.143 INFO:teuthology.orchestra.run.smithi083.stdout:HOME_URL="https://centos.org/" 2024-09-16T16:31:25.143 INFO:teuthology.orchestra.run.smithi083.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T16:31:25.143 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T16:31:25.143 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T16:31:25.144 INFO:teuthology.lock.ops:Updating smithi083.front.sepia.ceph.com on lock server 2024-09-16T16:31:25.161 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T16:31:25.169 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T16:31:25.175 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T16:31:25.175 DEBUG:teuthology.orchestra.run.smithi083:> test '!' -e /home/ubuntu/cephtest 2024-09-16T16:31:25.199 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T16:31:25.207 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T16:31:25.207 DEBUG:teuthology.orchestra.run.smithi083:> test -z $(ls -A /var/lib/ceph) 2024-09-16T16:31:25.257 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T16:31:25.289 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T16:31:25.303 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-09-16T16:31:25.303 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-09-16T16:31:25.304 DEBUG:teuthology.orchestra.run.smithi083:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T16:31:25.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T16:31:25.320 DEBUG:teuthology.orchestra.run.smithi083:> uname -r 2024-09-16T16:31:25.377 INFO:teuthology.orchestra.run.smithi083.stdout:5.14.0-503.el9.x86_64 2024-09-16T16:31:25.377 INFO:teuthology.task.kernel:Running kernel on smithi083: 5.14.0-503.el9.x86_64 2024-09-16T16:31:25.378 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum install -y kernel 2024-09-16T16:31:28.781 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.2 MB 00:01 2024-09-16T16:31:33.174 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - AppStream 6.2 MB/s | 20 MB 00:03 2024-09-16T16:31:37.449 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-16T16:31:39.316 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 19 kB 00:00 2024-09-16T16:31:41.104 INFO:teuthology.orchestra.run.smithi083.stdout:Extra Packages for Enterprise Linux 14 MB/s | 23 MB 00:01 2024-09-16T16:31:46.568 INFO:teuthology.orchestra.run.smithi083.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-09-16T16:31:48.575 INFO:teuthology.orchestra.run.smithi083.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T16:31:48.575 INFO:teuthology.orchestra.run.smithi083.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T16:31:48.668 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-16T16:31:48.675 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:31:48.676 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-09-16T16:31:48.676 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:31:48.676 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-16T16:31:48.676 INFO:teuthology.orchestra.run.smithi083.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T16:31:48.676 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-09-16T16:31:48.676 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T16:31:48.676 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T16:31:48.676 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T16:31:48.676 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:31:48.676 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-16T16:31:48.676 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:31:48.676 INFO:teuthology.orchestra.run.smithi083.stdout:Install 4 Packages 2024-09-16T16:31:48.677 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:31:48.677 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 79 M 2024-09-16T16:31:48.677 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 126 M 2024-09-16T16:31:48.677 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-16T16:31:49.003 INFO:teuthology.orchestra.run.smithi083.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 283 kB/s | 23 kB 00:00 2024-09-16T16:31:52.154 INFO:teuthology.orchestra.run.smithi083.stdout:(2/4): kernel-modules-core-5.14.0-505.el9.x86_6 9.1 MB/s | 29 MB 00:03 2024-09-16T16:31:53.262 INFO:teuthology.orchestra.run.smithi083.stdout:(3/4): kernel-core-5.14.0-505.el9.x86_64.rpm 3.6 MB/s | 16 MB 00:04 2024-09-16T16:31:53.829 INFO:teuthology.orchestra.run.smithi083.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 7.0 MB/s | 35 MB 00:04 2024-09-16T16:31:53.829 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:31:53.829 INFO:teuthology.orchestra.run.smithi083.stdout:Total 15 MB/s | 79 MB 00:05 2024-09-16T16:31:54.514 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-16T16:31:54.856 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-16T16:31:54.856 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-16T16:31:55.801 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-16T16:31:55.802 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-16T16:31:58.224 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-16T16:31:58.964 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T16:31:59.049 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T16:32:01.831 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T16:32:01.968 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T16:32:06.804 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T16:32:06.835 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T16:32:11.669 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T16:32:49.850 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T16:32:49.889 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T16:32:50.240 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T16:32:50.240 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T16:32:50.240 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T16:32:50.240 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T16:32:51.097 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T16:32:51.097 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:32:51.097 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-16T16:32:51.097 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T16:32:51.097 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T16:32:51.097 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T16:32:51.097 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T16:32:51.097 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:32:51.097 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-16T16:32:51.469 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T16:32:51.551 INFO:teuthology.orchestra.run.smithi083.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T16:32:51.551 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T16:32:51.551 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T16:32:51.552 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T16:32:51.552 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T16:32:51.552 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T16:32:51.552 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi083.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T16:32:51.552 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T16:32:51.611 INFO:teuthology.orchestra.run.smithi083.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T16:32:51.611 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T16:32:51.612 DEBUG:teuthology.orchestra.run.smithi083:> sudo rpm -qi grub2-tools 2024-09-16T16:32:51.643 INFO:teuthology.orchestra.run.smithi083.stdout:Name : grub2-tools 2024-09-16T16:32:51.644 INFO:teuthology.orchestra.run.smithi083.stdout:Epoch : 1 2024-09-16T16:32:51.644 INFO:teuthology.orchestra.run.smithi083.stdout:Version : 2.06 2024-09-16T16:32:51.644 INFO:teuthology.orchestra.run.smithi083.stdout:Release : 82.el9 2024-09-16T16:32:51.644 INFO:teuthology.orchestra.run.smithi083.stdout:Architecture: x86_64 2024-09-16T16:32:51.644 INFO:teuthology.orchestra.run.smithi083.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T16:32:51.644 INFO:teuthology.orchestra.run.smithi083.stdout:Group : Unspecified 2024-09-16T16:32:51.644 INFO:teuthology.orchestra.run.smithi083.stdout:Size : 8274670 2024-09-16T16:32:51.644 INFO:teuthology.orchestra.run.smithi083.stdout:License : GPLv3+ 2024-09-16T16:32:51.644 INFO:teuthology.orchestra.run.smithi083.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T16:32:51.644 INFO:teuthology.orchestra.run.smithi083.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T16:32:51.644 INFO:teuthology.orchestra.run.smithi083.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T16:32:51.645 INFO:teuthology.orchestra.run.smithi083.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T16:32:51.645 INFO:teuthology.orchestra.run.smithi083.stdout:Packager : builder@centos.org 2024-09-16T16:32:51.645 INFO:teuthology.orchestra.run.smithi083.stdout:Vendor : CentOS 2024-09-16T16:32:51.645 INFO:teuthology.orchestra.run.smithi083.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T16:32:51.645 INFO:teuthology.orchestra.run.smithi083.stdout:Summary : Support tools for GRUB. 2024-09-16T16:32:51.645 INFO:teuthology.orchestra.run.smithi083.stdout:Description : 2024-09-16T16:32:51.645 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:32:51.645 INFO:teuthology.orchestra.run.smithi083.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T16:32:51.645 INFO:teuthology.orchestra.run.smithi083.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T16:32:51.645 INFO:teuthology.orchestra.run.smithi083.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T16:32:51.645 INFO:teuthology.orchestra.run.smithi083.stdout:hardware devices. 2024-09-16T16:32:51.645 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:32:51.645 INFO:teuthology.orchestra.run.smithi083.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T16:32:51.646 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T16:32:51.646 INFO:teuthology.task.kernel:Updating grub on smithi083 to boot 5.14.0-505.el9.x86_64 2024-09-16T16:32:51.647 DEBUG:teuthology.orchestra.run.smithi083:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T16:32:52.322 INFO:teuthology.orchestra.run.smithi083.stderr:Generating grub configuration file ... 2024-09-16T16:32:53.563 INFO:teuthology.orchestra.run.smithi083.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T16:32:53.590 INFO:teuthology.orchestra.run.smithi083.stderr:done 2024-09-16T16:32:53.592 DEBUG:teuthology.orchestra.run.smithi083:> mktemp 2024-09-16T16:32:53.607 INFO:teuthology.orchestra.run.smithi083.stdout:/tmp/tmp.p7XfgCqTrV 2024-09-16T16:32:53.608 DEBUG:teuthology.orchestra.run.smithi083:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.p7XfgCqTrV 2024-09-16T16:32:53.674 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0666 /tmp/tmp.p7XfgCqTrV 2024-09-16T16:32:53.826 DEBUG:teuthology.orchestra.remote:smithi083:/tmp/tmp.p7XfgCqTrV is 6KB 2024-09-16T16:32:53.874 DEBUG:teuthology.orchestra.run.smithi083:> rm -fr /tmp/tmp.p7XfgCqTrV 2024-09-16T16:32:53.889 DEBUG:teuthology.orchestra.run.smithi083:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T16:32:53.952 INFO:teuthology.orchestra.run.smithi083.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T16:32:53.952 INFO:teuthology.orchestra.run.smithi083.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T16:32:53.952 INFO:teuthology.orchestra.run.smithi083.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T16:32:53.952 INFO:teuthology.orchestra.run.smithi083.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T16:32:53.953 DEBUG:teuthology.orchestra.run.smithi083:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T16:32:54.120 DEBUG:teuthology.orchestra.run.smithi083:> sudo shutdown -r now 2024-09-16T16:33:24.153 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T16:33:24.153 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-09-16T16:33:24.154 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T16:33:42.717 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2024-09-16T16:33:51.725 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-09-16T16:33:51.726 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T16:33:54.781 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2024-09-16T16:34:06.793 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-09-16T16:34:06.793 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T16:34:25.213 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2024-09-16T16:34:40.223 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-09-16T16:34:40.224 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T16:34:40.607 DEBUG:teuthology.orchestra.run.smithi083:> true 2024-09-16T16:34:41.045 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-09-16T16:34:41.045 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-16T16:34:41.046 DEBUG:teuthology.orchestra.run.smithi083:> uname -r 2024-09-16T16:34:41.060 INFO:teuthology.orchestra.run.smithi083.stdout:5.14.0-505.el9.x86_64 2024-09-16T16:34:41.061 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T16:34:41.061 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T16:34:41.061 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T16:34:42.061 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T16:34:42.062 DEBUG:teuthology.orchestra.run.smithi083:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T16:34:42.129 INFO:teuthology.orchestra.run.smithi083.stdout:ttyS1 2024-09-16T16:34:42.151 DEBUG:teuthology.parallel:result is None 2024-09-16T16:34:42.152 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T16:34:42.159 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T16:34:42.160 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T16:34:42.175 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T16:34:42.184 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T16:34:42.192 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T16:34:42.193 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T16:34:42.267 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T16:34:42.275 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T16:34:42.275 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T16:34:42.305 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:34:42.314 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:34:42.316 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T16:34:42.326 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T16:34:42.326 DEBUG:teuthology.orchestra.run.smithi083:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T16:34:42.380 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T16:34:42.393 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T16:34:42.393 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T16:34:42.436 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T16:34:42.520 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T16:34:42.564 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T16:34:42.644 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T16:34:42.686 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T16:34:42.686 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T16:34:42.750 DEBUG:teuthology.orchestra.run.smithi083:> sudo service rsyslog restart 2024-09-16T16:34:42.836 INFO:teuthology.orchestra.run.smithi083.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T16:34:43.212 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T16:34:43.220 INFO:teuthology.task.internal:Starting timer... 2024-09-16T16:34:43.220 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T16:34:43.230 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T16:34:43.238 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-16T16:34:43.238 DEBUG:teuthology.orchestra.run.smithi083:> sudo service auditd rotate 2024-09-16T16:34:43.278 INFO:teuthology.orchestra.run.smithi083.stdout:Rotating logs: 2024-09-16T16:34:43.280 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T16:34:43.280 DEBUG:teuthology.orchestra.run.smithi083:> /usr/sbin/getenforce 2024-09-16T16:34:43.359 INFO:teuthology.orchestra.run.smithi083.stdout:Permissive 2024-09-16T16:34:43.359 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi083.front.sepia.ceph.com': 'permissive'} 2024-09-16T16:34:43.360 DEBUG:teuthology.orchestra.run.smithi083:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T16:34:43.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T16:34:43.425 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T16:34:43.426 DEBUG:teuthology.orchestra.run.smithi083:> sudo /usr/sbin/setenforce permissive 2024-09-16T16:34:43.501 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T16:34:43.510 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-16T16:34:43.513 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-16T16:34:43.620 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T16:34:43.631 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T16:34:43.632 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi083.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T16:39:00.667 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi083.front.sepia.ceph.com')] 2024-09-16T16:39:00.668 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-09-16T16:39:00.669 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T16:39:00.762 DEBUG:teuthology.orchestra.run.smithi083:> true 2024-09-16T16:39:00.846 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-09-16T16:39:00.846 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T16:39:00.855 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T16:39:00.856 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T16:39:00.856 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T16:39:00.919 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T16:39:00.936 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T16:39:00.961 INFO:teuthology.orchestra.run.smithi083.stderr:sudo: ntpd: command not found 2024-09-16T16:39:00.973 INFO:teuthology.orchestra.run.smithi083.stdout:506 Cannot talk to daemon 2024-09-16T16:39:00.988 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T16:39:01.002 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T16:39:01.062 INFO:teuthology.orchestra.run.smithi083.stderr:bash: line 1: ntpq: command not found 2024-09-16T16:39:01.070 INFO:teuthology.orchestra.run.smithi083.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T16:39:01.070 INFO:teuthology.orchestra.run.smithi083.stdout:=============================================================================== 2024-09-16T16:39:01.070 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T16:39:01.070 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T16:39:01.070 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T16:39:01.070 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T16:39:01.070 INFO:teuthology.run_tasks:Running task install... 2024-09-16T16:39:01.080 DEBUG:teuthology.task.install:project ceph 2024-09-16T16:39:01.080 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '81fedcb3b484c843d22ffbc8d179cb567b7083b5'}} 2024-09-16T16:39:01.080 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '81fedcb3b484c843d22ffbc8d179cb567b7083b5'} 2024-09-16T16:39:01.081 INFO:teuthology.task.install:Using flavor: crimson 2024-09-16T16:39:01.090 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T16:39:01.090 INFO:teuthology.task.install:extra packages: [] 2024-09-16T16:39:01.091 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81fedcb3b484c843d22ffbc8d179cb567b7083b5', 'tag': None, 'wait_for_package': False} 2024-09-16T16:39:01.091 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=81fedcb3b484c843d22ffbc8d179cb567b7083b5 2024-09-16T16:39:01.245 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-59543/81fedcb3b484c843d22ffbc8d179cb567b7083b5/centos/9/flavors/crimson/ 2024-09-16T16:39:01.246 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4879.g81fedcb3 2024-09-16T16:39:01.373 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-59543/81fedcb3b484c843d22ffbc8d179cb567b7083b5/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-59543/81fedcb3b484c843d22ffbc8d179cb567b7083b5/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-59543/81fedcb3b484c843d22ffbc8d179cb567b7083b5/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T16:39:01.377 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T16:39:01.377 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T16:39:01.448 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T16:39:01.449 DEBUG:teuthology.orchestra.run.smithi083:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/81fedcb3b484c843d22ffbc8d179cb567b7083b5/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T16:39:01.514 DEBUG:teuthology.orchestra.run.smithi083:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T16:39:01.594 DEBUG:teuthology.orchestra.run.smithi083:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T16:39:01.658 INFO:teuthology.orchestra.run.smithi083.stdout:check_obsoletes = 1 2024-09-16T16:39:01.659 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum clean all 2024-09-16T16:39:01.901 INFO:teuthology.orchestra.run.smithi083.stdout:45 files removed 2024-09-16T16:39:01.925 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-radosgw 2024-09-16T16:39:02.473 INFO:teuthology.orchestra.run.smithi083.stdout:ceph packages for x86_64 334 kB/s | 88 kB 00:00 2024-09-16T16:39:02.766 INFO:teuthology.orchestra.run.smithi083.stdout:ceph noarch packages 97 kB/s | 24 kB 00:00 2024-09-16T16:39:03.041 INFO:teuthology.orchestra.run.smithi083.stdout:ceph source packages 8.7 kB/s | 2.1 kB 00:00 2024-09-16T16:39:03.508 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.2 MB 00:00 2024-09-16T16:39:09.018 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - AppStream 4.6 MB/s | 20 MB 00:04 2024-09-16T16:39:13.336 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-16T16:39:15.619 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - Extras packages 17 kB/s | 19 kB 00:01 2024-09-16T16:39:16.941 INFO:teuthology.orchestra.run.smithi083.stdout:Extra Packages for Enterprise Linux 19 MB/s | 23 MB 00:01 2024-09-16T16:39:22.880 INFO:teuthology.orchestra.run.smithi083.stdout:lab-extras 2.7 kB/s | 1.7 kB 00:00 2024-09-16T16:39:25.232 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-16T16:39:25.235 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:39:25.235 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-09-16T16:39:25.235 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:39:25.235 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-16T16:39:25.235 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 10 M 2024-09-16T16:39:25.235 INFO:teuthology.orchestra.run.smithi083.stdout:Upgrading: 2024-09-16T16:39:25.235 INFO:teuthology.orchestra.run.smithi083.stdout: librados2 x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 3.5 M 2024-09-16T16:39:25.235 INFO:teuthology.orchestra.run.smithi083.stdout: librbd1 x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 3.8 M 2024-09-16T16:39:25.236 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-09-16T16:39:25.236 INFO:teuthology.orchestra.run.smithi083.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T16:39:25.236 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 6.1 M 2024-09-16T16:39:25.236 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 20 M 2024-09-16T16:39:25.236 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-selinux x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 25 k 2024-09-16T16:39:25.236 INFO:teuthology.orchestra.run.smithi083.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T16:39:25.236 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T16:39:25.236 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T16:39:25.236 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2 x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 809 k 2024-09-16T16:39:25.236 INFO:teuthology.orchestra.run.smithi083.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-16T16:39:25.236 INFO:teuthology.orchestra.run.smithi083.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T16:39:25.237 INFO:teuthology.orchestra.run.smithi083.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T16:39:25.237 INFO:teuthology.orchestra.run.smithi083.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T16:39:25.237 INFO:teuthology.orchestra.run.smithi083.stdout: libradosstriper1 x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 558 k 2024-09-16T16:39:25.237 INFO:teuthology.orchestra.run.smithi083.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T16:39:25.237 INFO:teuthology.orchestra.run.smithi083.stdout: librgw2 x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 5.4 M 2024-09-16T16:39:25.237 INFO:teuthology.orchestra.run.smithi083.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T16:39:25.237 INFO:teuthology.orchestra.run.smithi083.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T16:39:25.237 INFO:teuthology.orchestra.run.smithi083.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T16:39:25.237 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-argparse x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 45 k 2024-09-16T16:39:25.237 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-common x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 143 k 2024-09-16T16:39:25.237 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 160 k 2024-09-16T16:39:25.237 INFO:teuthology.orchestra.run.smithi083.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T16:39:25.237 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 315 k 2024-09-16T16:39:25.238 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 302 k 2024-09-16T16:39:25.238 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 99 k 2024-09-16T16:39:25.238 INFO:teuthology.orchestra.run.smithi083.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T16:39:25.238 INFO:teuthology.orchestra.run.smithi083.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T16:39:25.238 INFO:teuthology.orchestra.run.smithi083.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T16:39:25.238 INFO:teuthology.orchestra.run.smithi083.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T16:39:25.238 INFO:teuthology.orchestra.run.smithi083.stdout:Installing weak dependencies: 2024-09-16T16:39:25.238 INFO:teuthology.orchestra.run.smithi083.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T16:39:25.238 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:39:25.238 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-16T16:39:25.238 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:39:25.238 INFO:teuthology.orchestra.run.smithi083.stdout:Install 31 Packages 2024-09-16T16:39:25.238 INFO:teuthology.orchestra.run.smithi083.stdout:Upgrade 2 Packages 2024-09-16T16:39:25.239 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:39:25.239 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 60 M 2024-09-16T16:39:25.239 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-16T16:39:26.123 INFO:teuthology.orchestra.run.smithi083.stdout:(1/33): ceph-base-19.3.0-4879.g81fedcb3.el9.x86 12 MB/s | 6.1 MB 00:00 2024-09-16T16:39:26.181 INFO:teuthology.orchestra.run.smithi083.stdout:(2/33): ceph-selinux-19.3.0-4879.g81fedcb3.el9. 432 kB/s | 25 kB 00:00 2024-09-16T16:39:26.273 INFO:teuthology.orchestra.run.smithi083.stdout:(3/33): libcephfs2-19.3.0-4879.g81fedcb3.el9.x8 8.6 MB/s | 809 kB 00:00 2024-09-16T16:39:26.357 INFO:teuthology.orchestra.run.smithi083.stdout:(4/33): libradosstriper1-19.3.0-4879.g81fedcb3. 6.5 MB/s | 558 kB 00:00 2024-09-16T16:39:26.482 INFO:teuthology.orchestra.run.smithi083.stdout:(5/33): ceph-radosgw-19.3.0-4879.g81fedcb3.el9. 12 MB/s | 10 MB 00:00 2024-09-16T16:39:26.541 INFO:teuthology.orchestra.run.smithi083.stdout:(6/33): python3-ceph-argparse-19.3.0-4879.g81fe 773 kB/s | 45 kB 00:00 2024-09-16T16:39:26.641 INFO:teuthology.orchestra.run.smithi083.stdout:(7/33): librgw2-19.3.0-4879.g81fedcb3.el9.x86_6 19 MB/s | 5.4 MB 00:00 2024-09-16T16:39:26.666 INFO:teuthology.orchestra.run.smithi083.stdout:(8/33): python3-ceph-common-19.3.0-4879.g81fedc 1.1 MB/s | 143 kB 00:00 2024-09-16T16:39:26.692 INFO:teuthology.orchestra.run.smithi083.stdout:(9/33): python3-cephfs-19.3.0-4879.g81fedcb3.el 3.1 MB/s | 160 kB 00:00 2024-09-16T16:39:26.876 INFO:teuthology.orchestra.run.smithi083.stdout:(10/33): ceph-common-19.3.0-4879.g81fedcb3.el9. 16 MB/s | 20 MB 00:01 2024-09-16T16:39:26.901 INFO:teuthology.orchestra.run.smithi083.stdout:(11/33): python3-rados-19.3.0-4879.g81fedcb3.el 1.3 MB/s | 315 kB 00:00 2024-09-16T16:39:26.927 INFO:teuthology.orchestra.run.smithi083.stdout:(12/33): python3-rgw-19.3.0-4879.g81fedcb3.el9. 1.9 MB/s | 99 kB 00:00 2024-09-16T16:39:26.954 INFO:teuthology.orchestra.run.smithi083.stdout:(13/33): python3-rbd-19.3.0-4879.g81fedcb3.el9. 1.1 MB/s | 302 kB 00:00 2024-09-16T16:39:27.218 INFO:teuthology.orchestra.run.smithi083.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 337 kB/s | 107 kB 00:00 2024-09-16T16:39:27.244 INFO:teuthology.orchestra.run.smithi083.stdout:(15/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 549 kB/s | 160 kB 00:00 2024-09-16T16:39:27.269 INFO:teuthology.orchestra.run.smithi083.stdout:(16/33): libnbd-1.20.2-2.el9.x86_64.rpm 498 kB/s | 171 kB 00:00 2024-09-16T16:39:27.294 INFO:teuthology.orchestra.run.smithi083.stdout:(17/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 599 kB/s | 45 kB 00:00 2024-09-16T16:39:27.386 INFO:teuthology.orchestra.run.smithi083.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 462 kB/s | 42 kB 00:00 2024-09-16T16:39:27.411 INFO:teuthology.orchestra.run.smithi083.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.0 MB/s | 292 kB 00:00 2024-09-16T16:39:27.453 INFO:teuthology.orchestra.run.smithi083.stdout:(20/33): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 662 kB 00:00 2024-09-16T16:39:27.487 INFO:teuthology.orchestra.run.smithi083.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 475 kB/s | 36 kB 00:00 2024-09-16T16:39:27.529 INFO:teuthology.orchestra.run.smithi083.stdout:(22/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 630 kB/s | 47 kB 00:00 2024-09-16T16:39:27.554 INFO:teuthology.orchestra.run.smithi083.stdout:(23/33): qatlib-24.02.0-1.el9.x86_64.rpm 1.3 MB/s | 221 kB 00:00 2024-09-16T16:39:27.604 INFO:teuthology.orchestra.run.smithi083.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.6 MB/s | 308 kB 00:00 2024-09-16T16:39:27.630 INFO:teuthology.orchestra.run.smithi083.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 2.0 MB/s | 49 kB 00:00 2024-09-16T16:39:27.655 INFO:teuthology.orchestra.run.smithi083.stdout:(26/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 280 kB/s | 26 kB 00:00 2024-09-16T16:39:27.680 INFO:teuthology.orchestra.run.smithi083.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-16T16:39:27.714 INFO:teuthology.orchestra.run.smithi083.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.7 MB/s | 191 kB 00:00 2024-09-16T16:39:27.773 INFO:teuthology.orchestra.run.smithi083.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-09-16T16:39:27.848 INFO:teuthology.orchestra.run.smithi083.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-09-16T16:39:27.957 INFO:teuthology.orchestra.run.smithi083.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-09-16T16:39:28.373 INFO:teuthology.orchestra.run.smithi083.stdout:(32/33): librbd1-19.3.0-4879.g81fedcb3.el9.x86_ 7.2 MB/s | 3.8 MB 00:00 2024-09-16T16:39:28.682 INFO:teuthology.orchestra.run.smithi083.stdout:(33/33): librados2-19.3.0-4879.g81fedcb3.el9.x8 3.8 MB/s | 3.5 MB 00:00 2024-09-16T16:39:28.688 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:39:28.688 INFO:teuthology.orchestra.run.smithi083.stdout:Total 18 MB/s | 60 MB 00:03 2024-09-16T16:39:28.804 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-16T16:39:28.843 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-16T16:39:28.843 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-16T16:39:29.509 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-16T16:39:29.510 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-16T16:39:31.163 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-16T16:39:31.668 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-16T16:39:31.742 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : librados2-2:19.3.0-4879.g81fedcb3.el9.x86_64 2/35 2024-09-16T16:39:31.843 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados2-2:19.3.0-4879.g81fedcb3.el9.x86_64 2/35 2024-09-16T16:39:31.942 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rados-2:19.3.0-4879.g81fedcb3.el9.x86_64 3/35 2024-09-16T16:39:31.973 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephfs2-2:19.3.0-4879.g81fedcb3.el9.x86_64 4/35 2024-09-16T16:39:32.032 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephfs2-2:19.3.0-4879.g81fedcb3.el9.x86_64 4/35 2024-09-16T16:39:32.079 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/35 2024-09-16T16:39:32.117 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/35 2024-09-16T16:39:32.141 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/35 2024-09-16T16:39:32.164 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-16T16:39:32.191 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-16T16:39:32.383 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-16T16:39:32.422 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-16T16:39:32.457 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/35 2024-09-16T16:39:32.501 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ceph-argparse-2:19.3.0-4879.g81fedcb3.el9. 11/35 2024-09-16T16:39:32.530 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cephfs-2:19.3.0-4879.g81fedcb3.el9.x86_64 12/35 2024-09-16T16:39:32.619 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 13/35 2024-09-16T16:39:32.643 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libradosstriper1-2:19.3.0-4879.g81fedcb3.el9.x86_6 14/35 2024-09-16T16:39:32.754 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4879.g81fedcb3.el9.x86_6 14/35 2024-09-16T16:39:32.784 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : thrift-0.15.0-2.el9.x86_64 15/35 2024-09-16T16:39:32.896 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : re2-1:20211101-20.el9.x86_64 16/35 2024-09-16T16:39:32.927 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 17/35 2024-09-16T16:39:32.983 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 18/35 2024-09-16T16:39:33.016 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 19/35 2024-09-16T16:39:33.046 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 20/35 2024-09-16T16:39:33.082 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 21/35 2024-09-16T16:39:33.450 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-16T16:39:33.500 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : librbd1-2:19.3.0-4879.g81fedcb3.el9.x86_64 23/35 2024-09-16T16:39:33.591 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librbd1-2:19.3.0-4879.g81fedcb3.el9.x86_64 23/35 2024-09-16T16:39:33.621 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 24/35 2024-09-16T16:39:33.657 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-16T16:39:34.234 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-16T16:39:34.261 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librgw2-2:19.3.0-4879.g81fedcb3.el9.x86_64 27/35 2024-09-16T16:39:34.333 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librgw2-2:19.3.0-4879.g81fedcb3.el9.x86_64 27/35 2024-09-16T16:39:34.413 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 28/35 2024-09-16T16:39:34.446 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ceph-common-2:19.3.0-4879.g81fedcb3.el9.x8 29/35 2024-09-16T16:39:36.936 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-common-2:19.3.0-4879.g81fedcb3.el9.x86_64 30/35 2024-09-16T16:39:36.984 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-common-2:19.3.0-4879.g81fedcb3.el9.x86_64 30/35 2024-09-16T16:39:37.683 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-common-2:19.3.0-4879.g81fedcb3.el9.x86_64 30/35 2024-09-16T16:39:37.730 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-base-2:19.3.0-4879.g81fedcb3.el9.x86_64 31/35 2024-09-16T16:39:37.825 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-base-2:19.3.0-4879.g81fedcb3.el9.x86_64 31/35 2024-09-16T16:39:37.826 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T16:39:37.826 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T16:39:37.826 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:39:37.881 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-selinux-2:19.3.0-4879.g81fedcb3.el9.x86_64 32/35 2024-09-16T16:39:48.347 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4879.g81fedcb3.el9.x86_64 32/35 2024-09-16T16:39:48.347 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /sys 2024-09-16T16:39:48.347 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /proc 2024-09-16T16:39:48.347 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /mnt 2024-09-16T16:39:48.347 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /var/tmp 2024-09-16T16:39:48.348 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /home 2024-09-16T16:39:48.348 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /root 2024-09-16T16:39:48.348 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /tmp 2024-09-16T16:39:48.348 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:39:49.477 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-radosgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 33/35 2024-09-16T16:39:49.503 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 33/35 2024-09-16T16:39:49.503 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T16:39:49.503 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T16:39:49.503 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T16:39:49.503 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T16:39:49.503 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:39:49.504 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-16T16:39:49.596 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-16T16:39:49.596 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T16:39:50.937 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T16:39:50.937 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-base-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/35 2024-09-16T16:39:50.938 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-common-2:19.3.0-4879.g81fedcb3.el9.x86_64 2/35 2024-09-16T16:39:50.938 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-radosgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 3/35 2024-09-16T16:39:50.938 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-selinux-2:19.3.0-4879.g81fedcb3.el9.x86_64 4/35 2024-09-16T16:39:50.938 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephfs2-2:19.3.0-4879.g81fedcb3.el9.x86_64 5/35 2024-09-16T16:39:50.938 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libradosstriper1-2:19.3.0-4879.g81fedcb3.el9.x86_6 6/35 2024-09-16T16:39:50.938 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librgw2-2:19.3.0-4879.g81fedcb3.el9.x86_64 7/35 2024-09-16T16:39:50.938 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4879.g81fedcb3.el9. 8/35 2024-09-16T16:39:50.938 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ceph-common-2:19.3.0-4879.g81fedcb3.el9.x8 9/35 2024-09-16T16:39:50.938 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cephfs-2:19.3.0-4879.g81fedcb3.el9.x86_64 10/35 2024-09-16T16:39:50.938 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rados-2:19.3.0-4879.g81fedcb3.el9.x86_64 11/35 2024-09-16T16:39:50.938 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 12/35 2024-09-16T16:39:50.938 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 13/35 2024-09-16T16:39:50.939 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-16T16:39:50.939 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-16T16:39:50.939 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-16T16:39:50.939 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-16T16:39:50.939 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-16T16:39:50.939 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-16T16:39:50.939 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-16T16:39:50.939 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-16T16:39:50.939 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-16T16:39:50.939 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-16T16:39:50.939 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-16T16:39:50.939 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-16T16:39:50.939 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-16T16:39:50.939 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-16T16:39:50.940 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-16T16:39:50.940 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-16T16:39:50.940 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-16T16:39:50.941 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-16T16:39:50.941 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados2-2:19.3.0-4879.g81fedcb3.el9.x86_64 32/35 2024-09-16T16:39:50.941 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-16T16:39:50.941 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librbd1-2:19.3.0-4879.g81fedcb3.el9.x86_64 34/35 2024-09-16T16:39:51.903 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T16:39:51.903 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:39:51.903 INFO:teuthology.orchestra.run.smithi083.stdout:Upgraded: 2024-09-16T16:39:51.903 INFO:teuthology.orchestra.run.smithi083.stdout: librados2-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:39:51.903 INFO:teuthology.orchestra.run.smithi083.stdout: librbd1-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:39:51.903 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-16T16:39:51.903 INFO:teuthology.orchestra.run.smithi083.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T16:39:51.904 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:39:51.904 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:39:51.904 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:39:51.904 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-selinux-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:39:51.904 INFO:teuthology.orchestra.run.smithi083.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T16:39:51.904 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T16:39:51.904 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T16:39:51.904 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:39:51.904 INFO:teuthology.orchestra.run.smithi083.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-16T16:39:51.904 INFO:teuthology.orchestra.run.smithi083.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T16:39:51.904 INFO:teuthology.orchestra.run.smithi083.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T16:39:51.904 INFO:teuthology.orchestra.run.smithi083.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T16:39:51.904 INFO:teuthology.orchestra.run.smithi083.stdout: libradosstriper1-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:39:51.905 INFO:teuthology.orchestra.run.smithi083.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T16:39:51.905 INFO:teuthology.orchestra.run.smithi083.stdout: librgw2-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:39:51.905 INFO:teuthology.orchestra.run.smithi083.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T16:39:51.905 INFO:teuthology.orchestra.run.smithi083.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T16:39:51.905 INFO:teuthology.orchestra.run.smithi083.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T16:39:51.905 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-argparse-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:39:51.905 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-common-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:39:51.905 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:39:51.905 INFO:teuthology.orchestra.run.smithi083.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T16:39:51.905 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:39:51.905 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:39:51.905 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:39:51.906 INFO:teuthology.orchestra.run.smithi083.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T16:39:51.906 INFO:teuthology.orchestra.run.smithi083.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T16:39:51.906 INFO:teuthology.orchestra.run.smithi083.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T16:39:51.906 INFO:teuthology.orchestra.run.smithi083.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T16:39:51.906 INFO:teuthology.orchestra.run.smithi083.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T16:39:51.906 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:39:51.906 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-16T16:39:52.286 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-test 2024-09-16T16:39:52.844 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:00:30 ago on Mon 16 Sep 2024 04:39:22 PM UTC. 2024-09-16T16:39:52.987 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-16T16:39:52.988 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:39:52.988 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-09-16T16:39:52.988 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:39:52.989 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-16T16:39:52.989 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 85 M 2024-09-16T16:39:52.989 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-09-16T16:39:52.989 INFO:teuthology.orchestra.run.smithi083.stdout: libasan8 x86_64 14.2.1-1.2.el9 appstream 513 k 2024-09-16T16:39:52.989 INFO:teuthology.orchestra.run.smithi083.stdout: libcephsqlite x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 175 k 2024-09-16T16:39:52.989 INFO:teuthology.orchestra.run.smithi083.stdout: libubsan x86_64 11.5.0-2.el9 appstream 183 k 2024-09-16T16:39:52.989 INFO:teuthology.orchestra.run.smithi083.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-09-16T16:39:52.989 INFO:teuthology.orchestra.run.smithi083.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T16:39:52.989 INFO:teuthology.orchestra.run.smithi083.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T16:39:52.989 INFO:teuthology.orchestra.run.smithi083.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T16:39:52.989 INFO:teuthology.orchestra.run.smithi083.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-09-16T16:39:52.989 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:39:52.990 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-16T16:39:52.990 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:39:52.990 INFO:teuthology.orchestra.run.smithi083.stdout:Install 9 Packages 2024-09-16T16:39:52.990 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:39:52.990 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 87 M 2024-09-16T16:39:52.990 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 797 M 2024-09-16T16:39:52.990 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-16T16:39:53.952 INFO:teuthology.orchestra.run.smithi083.stdout:(1/9): libcephsqlite-19.3.0-4879.g81fedcb3.el9. 848 kB/s | 175 kB 00:00 2024-09-16T16:39:54.394 INFO:teuthology.orchestra.run.smithi083.stdout:(2/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 145 kB/s | 94 kB 00:00 2024-09-16T16:39:54.511 INFO:teuthology.orchestra.run.smithi083.stdout:(3/9): libasan8-14.2.1-1.2.el9.x86_64.rpm 919 kB/s | 513 kB 00:00 2024-09-16T16:39:54.778 INFO:teuthology.orchestra.run.smithi083.stdout:(4/9): libubsan-11.5.0-2.el9.x86_64.rpm 477 kB/s | 183 kB 00:00 2024-09-16T16:39:54.945 INFO:teuthology.orchestra.run.smithi083.stdout:(5/9): protobuf-3.14.0-13.el9.x86_64.rpm 2.3 MB/s | 1.0 MB 00:00 2024-09-16T16:39:55.012 INFO:teuthology.orchestra.run.smithi083.stdout:(6/9): socat-1.7.4.1-6.el9.x86_64.rpm 1.3 MB/s | 304 kB 00:00 2024-09-16T16:39:55.062 INFO:teuthology.orchestra.run.smithi083.stdout:(7/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 2.4 MB/s | 122 kB 00:00 2024-09-16T16:39:55.087 INFO:teuthology.orchestra.run.smithi083.stdout:(8/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 448 kB/s | 64 kB 00:00 2024-09-16T16:39:57.054 INFO:teuthology.orchestra.run.smithi083.stdout:(9/9): ceph-test-19.3.0-4879.g81fedcb3.el9.x86_ 26 MB/s | 85 MB 00:03 2024-09-16T16:39:57.058 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:39:57.058 INFO:teuthology.orchestra.run.smithi083.stdout:Total 21 MB/s | 87 MB 00:04 2024-09-16T16:39:57.140 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-16T16:39:57.164 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-16T16:39:57.164 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-16T16:39:57.778 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-16T16:39:57.778 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-16T16:39:58.716 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-16T16:39:58.786 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-09-16T16:39:58.837 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-09-16T16:39:58.892 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/9 2024-09-16T16:39:58.919 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-09-16T16:39:58.950 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-16T16:39:58.974 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-16T16:39:59.058 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-16T16:39:59.099 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-09-16T16:39:59.126 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephsqlite-2:19.3.0-4879.g81fedcb3.el9.x86_64 8/9 2024-09-16T16:40:10.312 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4879.g81fedcb3.el9.x86_64 8/9 2024-09-16T16:40:10.401 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-test-2:19.3.0-4879.g81fedcb3.el9.x86_64 9/9 2024-09-16T16:40:16.742 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-test-2:19.3.0-4879.g81fedcb3.el9.x86_64 9/9 2024-09-16T16:40:16.742 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-test-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/9 2024-09-16T16:40:16.743 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephsqlite-2:19.3.0-4879.g81fedcb3.el9.x86_64 2/9 2024-09-16T16:40:16.743 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-09-16T16:40:16.743 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libasan8-14.2.1-1.2.el9.x86_64 4/9 2024-09-16T16:40:16.743 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-16T16:40:16.743 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-09-16T16:40:16.743 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 7/9 2024-09-16T16:40:16.743 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-09-16T16:40:17.176 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-09-16T16:40:17.176 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:40:17.176 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-16T16:40:17.177 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:40:17.177 INFO:teuthology.orchestra.run.smithi083.stdout: libasan8-14.2.1-1.2.el9.x86_64 2024-09-16T16:40:17.177 INFO:teuthology.orchestra.run.smithi083.stdout: libcephsqlite-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:40:17.177 INFO:teuthology.orchestra.run.smithi083.stdout: libubsan-11.5.0-2.el9.x86_64 2024-09-16T16:40:17.177 INFO:teuthology.orchestra.run.smithi083.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-09-16T16:40:17.177 INFO:teuthology.orchestra.run.smithi083.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T16:40:17.177 INFO:teuthology.orchestra.run.smithi083.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T16:40:17.177 INFO:teuthology.orchestra.run.smithi083.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T16:40:17.177 INFO:teuthology.orchestra.run.smithi083.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-09-16T16:40:17.177 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:40:17.177 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-16T16:40:17.543 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph 2024-09-16T16:40:18.110 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:00:56 ago on Mon 16 Sep 2024 04:39:22 PM UTC. 2024-09-16T16:40:18.246 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-16T16:40:18.249 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:40:18.249 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-09-16T16:40:18.249 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:40:18.249 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-16T16:40:18.249 INFO:teuthology.orchestra.run.smithi083.stdout: ceph x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 6.5 k 2024-09-16T16:40:18.249 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-09-16T16:40:18.249 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 2.2 M 2024-09-16T16:40:18.249 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 1.6 M 2024-09-16T16:40:18.250 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 291 k 2024-09-16T16:40:18.250 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 5.1 M 2024-09-16T16:40:18.250 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 70 M 2024-09-16T16:40:18.250 INFO:teuthology.orchestra.run.smithi083.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T16:40:18.250 INFO:teuthology.orchestra.run.smithi083.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T16:40:18.250 INFO:teuthology.orchestra.run.smithi083.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T16:40:18.250 INFO:teuthology.orchestra.run.smithi083.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T16:40:18.250 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T16:40:18.250 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T16:40:18.250 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T16:40:18.250 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T16:40:18.250 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T16:40:18.250 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T16:40:18.251 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-collections 2024-09-16T16:40:18.251 INFO:teuthology.orchestra.run.smithi083.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-16T16:40:18.251 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T16:40:18.251 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T16:40:18.251 INFO:teuthology.orchestra.run.smithi083.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T16:40:18.251 INFO:teuthology.orchestra.run.smithi083.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T16:40:18.251 INFO:teuthology.orchestra.run.smithi083.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T16:40:18.251 INFO:teuthology.orchestra.run.smithi083.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T16:40:18.251 INFO:teuthology.orchestra.run.smithi083.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T16:40:18.251 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T16:40:18.251 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T16:40:18.251 INFO:teuthology.orchestra.run.smithi083.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T16:40:18.251 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T16:40:18.252 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T16:40:18.252 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T16:40:18.252 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T16:40:18.252 INFO:teuthology.orchestra.run.smithi083.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T16:40:18.252 INFO:teuthology.orchestra.run.smithi083.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T16:40:18.252 INFO:teuthology.orchestra.run.smithi083.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T16:40:18.252 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T16:40:18.252 INFO:teuthology.orchestra.run.smithi083.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T16:40:18.252 INFO:teuthology.orchestra.run.smithi083.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T16:40:18.252 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:40:18.252 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-16T16:40:18.252 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:40:18.253 INFO:teuthology.orchestra.run.smithi083.stdout:Install 37 Packages 2024-09-16T16:40:18.253 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:40:18.253 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 84 M 2024-09-16T16:40:18.254 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 897 M 2024-09-16T16:40:18.254 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-16T16:40:19.610 INFO:teuthology.orchestra.run.smithi083.stdout:(1/37): ceph-19.3.0-4879.g81fedcb3.el9.x86_64.r 18 kB/s | 6.5 kB 00:00 2024-09-16T16:40:20.485 INFO:teuthology.orchestra.run.smithi083.stdout:(2/37): ceph-mds-19.3.0-4879.g81fedcb3.el9.x86_ 1.8 MB/s | 2.2 MB 00:01 2024-09-16T16:40:20.539 INFO:teuthology.orchestra.run.smithi083.stdout:(3/37): ceph-mgr-19.3.0-4879.g81fedcb3.el9.x86_ 1.2 MB/s | 1.6 MB 00:01 2024-09-16T16:40:20.708 INFO:teuthology.orchestra.run.smithi083.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4879.g81fe 1.7 MB/s | 291 kB 00:00 2024-09-16T16:40:20.856 INFO:teuthology.orchestra.run.smithi083.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 253 kB 00:00 2024-09-16T16:40:20.948 INFO:teuthology.orchestra.run.smithi083.stdout:(6/37): python3-cryptography-36.0.1-4.el9.x86_6 13 MB/s | 1.2 MB 00:00 2024-09-16T16:40:20.990 INFO:teuthology.orchestra.run.smithi083.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-16T16:40:21.040 INFO:teuthology.orchestra.run.smithi083.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 2.6 MB/s | 135 kB 00:00 2024-09-16T16:40:21.082 INFO:teuthology.orchestra.run.smithi083.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.9 MB/s | 125 kB 00:00 2024-09-16T16:40:21.124 INFO:teuthology.orchestra.run.smithi083.stdout:(10/37): python3-urllib3-1.26.5-6.el9.noarch.rp 5.0 MB/s | 215 kB 00:00 2024-09-16T16:40:21.241 INFO:teuthology.orchestra.run.smithi083.stdout:(11/37): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-09-16T16:40:21.283 INFO:teuthology.orchestra.run.smithi083.stdout:(12/37): python3-mako-1.1.4-6.el9.noarch.rpm 4.0 MB/s | 172 kB 00:00 2024-09-16T16:40:21.325 INFO:teuthology.orchestra.run.smithi083.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 838 kB/s | 35 kB 00:00 2024-09-16T16:40:21.369 INFO:teuthology.orchestra.run.smithi083.stdout:(14/37): python3-packaging-20.9-5.el9.noarch.rp 1.8 MB/s | 77 kB 00:00 2024-09-16T16:40:21.409 INFO:teuthology.orchestra.run.smithi083.stdout:(15/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-09-16T16:40:21.451 INFO:teuthology.orchestra.run.smithi083.stdout:(16/37): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-09-16T16:40:21.535 INFO:teuthology.orchestra.run.smithi083.stdout:(17/37): lua-devel-5.4.4-4.el9.x86_64.rpm 268 kB/s | 22 kB 00:00 2024-09-16T16:40:21.585 INFO:teuthology.orchestra.run.smithi083.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-09-16T16:40:21.610 INFO:teuthology.orchestra.run.smithi083.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2024-09-16T16:40:21.636 INFO:teuthology.orchestra.run.smithi083.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 6.7 MB/s | 172 kB 00:00 2024-09-16T16:40:21.669 INFO:teuthology.orchestra.run.smithi083.stdout:(21/37): python3-cherrypy-18.6.1-2.el9.noarch.r 10 MB/s | 358 kB 00:00 2024-09-16T16:40:21.695 INFO:teuthology.orchestra.run.smithi083.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 430 kB/s | 11 kB 00:00 2024-09-16T16:40:21.720 INFO:teuthology.orchestra.run.smithi083.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 706 kB/s | 18 kB 00:00 2024-09-16T16:40:21.745 INFO:teuthology.orchestra.run.smithi083.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 936 kB/s | 23 kB 00:00 2024-09-16T16:40:21.771 INFO:teuthology.orchestra.run.smithi083.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 775 kB/s | 19 kB 00:00 2024-09-16T16:40:21.796 INFO:teuthology.orchestra.run.smithi083.stdout:(26/37): python3-jaraco-text-3.2.0-6.el9.noarch 796 kB/s | 20 kB 00:00 2024-09-16T16:40:21.821 INFO:teuthology.orchestra.run.smithi083.stdout:(27/37): python3-logutils-0.3.5-21.el9.noarch.r 1.8 MB/s | 46 kB 00:00 2024-09-16T16:40:21.847 INFO:teuthology.orchestra.run.smithi083.stdout:(28/37): python3-more-itertools-8.12.0-2.el9.no 3.1 MB/s | 79 kB 00:00 2024-09-16T16:40:21.880 INFO:teuthology.orchestra.run.smithi083.stdout:(29/37): python3-pecan-1.4.2-3.el9.noarch.rpm 8.0 MB/s | 272 kB 00:00 2024-09-16T16:40:21.906 INFO:teuthology.orchestra.run.smithi083.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 651 kB/s | 16 kB 00:00 2024-09-16T16:40:21.931 INFO:teuthology.orchestra.run.smithi083.stdout:(31/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.5 MB/s | 90 kB 00:00 2024-09-16T16:40:21.956 INFO:teuthology.orchestra.run.smithi083.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-16T16:40:21.981 INFO:teuthology.orchestra.run.smithi083.stdout:(33/37): python3-webob-1.8.8-2.el9.noarch.rpm 9.1 MB/s | 230 kB 00:00 2024-09-16T16:40:22.015 INFO:teuthology.orchestra.run.smithi083.stdout:(34/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 12 MB/s | 427 kB 00:00 2024-09-16T16:40:22.040 INFO:teuthology.orchestra.run.smithi083.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 800 kB/s | 20 kB 00:00 2024-09-16T16:40:25.040 INFO:teuthology.orchestra.run.smithi083.stdout:(36/37): ceph-mon-19.3.0-4879.g81fedcb3.el9.x86 963 kB/s | 5.1 MB 00:05 2024-09-16T16:41:00.538 INFO:teuthology.orchestra.run.smithi083.stdout:(37/37): ceph-osd-19.3.0-4879.g81fedcb3.el9.x86 1.8 MB/s | 70 MB 00:40 2024-09-16T16:41:00.540 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:41:00.540 INFO:teuthology.orchestra.run.smithi083.stdout:Total 2.0 MB/s | 84 MB 00:42 2024-09-16T16:41:00.677 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-16T16:41:00.719 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-16T16:41:00.719 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-16T16:41:01.532 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-16T16:41:01.532 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-16T16:41:03.190 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-16T16:41:03.256 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-16T16:41:03.289 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-16T16:41:03.317 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-16T16:41:03.389 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-16T16:41:03.432 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-16T16:41:03.465 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-16T16:41:03.557 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-16T16:41:03.635 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-16T16:41:03.676 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-16T16:41:03.711 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-16T16:41:03.749 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-16T16:41:03.786 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-16T16:41:03.820 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-16T16:41:03.902 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-16T16:41:03.946 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-16T16:41:03.980 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-16T16:41:04.029 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-16T16:41:04.060 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-16T16:41:04.133 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-16T16:41:04.272 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-16T16:41:04.351 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-16T16:41:04.405 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-16T16:41:04.452 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-16T16:41:04.500 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-16T16:41:04.556 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-16T16:41:04.744 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-16T16:41:04.797 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-16T16:41:04.875 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-16T16:41:04.972 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-16T16:41:05.024 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-16T16:41:05.157 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-16T16:41:05.352 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4879.g81fedcb3.el9. 32/37 2024-09-16T16:41:05.376 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-2:19.3.0-4879.g81fedcb3.el9.x86_64 33/37 2024-09-16T16:41:05.401 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4879.g81fedcb3.el9.x86_64 33/37 2024-09-16T16:41:05.402 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T16:41:05.402 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T16:41:05.402 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T16:41:05.402 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T16:41:05.402 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:41:15.960 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-osd-2:19.3.0-4879.g81fedcb3.el9.x86_64 34/37 2024-09-16T16:41:15.989 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-osd-2:19.3.0-4879.g81fedcb3.el9.x86_64 34/37 2024-09-16T16:41:15.989 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T16:41:15.989 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T16:41:15.989 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T16:41:15.989 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T16:41:15.989 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:41:16.544 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mon-2:19.3.0-4879.g81fedcb3.el9.x86_64 35/37 2024-09-16T16:41:16.568 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mon-2:19.3.0-4879.g81fedcb3.el9.x86_64 35/37 2024-09-16T16:41:16.568 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T16:41:16.568 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T16:41:16.568 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T16:41:16.568 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T16:41:16.568 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:41:16.811 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mds-2:19.3.0-4879.g81fedcb3.el9.x86_64 36/37 2024-09-16T16:41:16.835 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mds-2:19.3.0-4879.g81fedcb3.el9.x86_64 36/37 2024-09-16T16:41:16.835 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T16:41:16.835 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T16:41:16.835 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T16:41:16.835 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T16:41:16.835 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:41:16.900 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-2:19.3.0-4879.g81fedcb3.el9.x86_64 37/37 2024-09-16T16:41:23.610 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-2:19.3.0-4879.g81fedcb3.el9.x86_64 37/37 2024-09-16T16:41:23.610 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/37 2024-09-16T16:41:23.610 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mds-2:19.3.0-4879.g81fedcb3.el9.x86_64 2/37 2024-09-16T16:41:23.610 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-2:19.3.0-4879.g81fedcb3.el9.x86_64 3/37 2024-09-16T16:41:23.610 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mon-2:19.3.0-4879.g81fedcb3.el9.x86_64 4/37 2024-09-16T16:41:23.610 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-osd-2:19.3.0-4879.g81fedcb3.el9.x86_64 5/37 2024-09-16T16:41:23.610 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4879.g81fedcb3.el9. 6/37 2024-09-16T16:41:23.610 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-16T16:41:23.610 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-16T16:41:23.611 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-16T16:41:23.611 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-16T16:41:23.611 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-16T16:41:23.611 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-16T16:41:23.611 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-16T16:41:23.611 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-16T16:41:23.611 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-16T16:41:23.611 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-16T16:41:23.611 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-16T16:41:23.612 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-16T16:41:23.612 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-16T16:41:23.612 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-16T16:41:23.612 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-16T16:41:23.612 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-16T16:41:23.612 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-16T16:41:23.612 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-16T16:41:23.612 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-16T16:41:23.612 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-16T16:41:23.613 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-16T16:41:23.614 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-16T16:41:23.614 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-16T16:41:23.614 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-16T16:41:23.614 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-16T16:41:23.614 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-16T16:41:23.614 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-16T16:41:23.614 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-16T16:41:23.614 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-16T16:41:23.614 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-16T16:41:24.592 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-16T16:41:24.592 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:41:24.592 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-16T16:41:24.592 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:41:24.593 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:41:24.593 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:41:24.593 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-modules-core-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T16:41:24.593 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:41:24.593 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:41:24.593 INFO:teuthology.orchestra.run.smithi083.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T16:41:24.593 INFO:teuthology.orchestra.run.smithi083.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T16:41:24.593 INFO:teuthology.orchestra.run.smithi083.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T16:41:24.593 INFO:teuthology.orchestra.run.smithi083.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T16:41:24.593 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T16:41:24.593 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T16:41:24.593 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T16:41:24.594 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T16:41:24.594 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T16:41:24.594 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T16:41:24.594 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T16:41:24.594 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T16:41:24.594 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T16:41:24.594 INFO:teuthology.orchestra.run.smithi083.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T16:41:24.594 INFO:teuthology.orchestra.run.smithi083.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T16:41:24.594 INFO:teuthology.orchestra.run.smithi083.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T16:41:24.594 INFO:teuthology.orchestra.run.smithi083.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T16:41:24.594 INFO:teuthology.orchestra.run.smithi083.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T16:41:24.594 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T16:41:24.594 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T16:41:24.595 INFO:teuthology.orchestra.run.smithi083.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T16:41:24.595 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T16:41:24.595 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T16:41:24.595 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T16:41:24.595 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T16:41:24.595 INFO:teuthology.orchestra.run.smithi083.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T16:41:24.595 INFO:teuthology.orchestra.run.smithi083.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T16:41:24.595 INFO:teuthology.orchestra.run.smithi083.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T16:41:24.595 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T16:41:24.595 INFO:teuthology.orchestra.run.smithi083.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T16:41:24.595 INFO:teuthology.orchestra.run.smithi083.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T16:41:24.595 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:41:24.595 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-16T16:41:24.889 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-base 2024-09-16T16:41:25.444 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:03 ago on Mon 16 Sep 2024 04:39:22 PM UTC. 2024-09-16T16:41:25.533 INFO:teuthology.orchestra.run.smithi083.stdout:Package ceph-base-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T16:41:25.575 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-16T16:41:25.576 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-09-16T16:41:25.576 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-16T16:41:25.662 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install cephadm 2024-09-16T16:41:26.219 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:04 ago on Mon 16 Sep 2024 04:39:22 PM UTC. 2024-09-16T16:41:26.352 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-16T16:41:26.352 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:41:26.352 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-09-16T16:41:26.353 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:41:26.353 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-16T16:41:26.353 INFO:teuthology.orchestra.run.smithi083.stdout: cephadm noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 785 k 2024-09-16T16:41:26.353 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:41:26.353 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-16T16:41:26.353 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:41:26.353 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-09-16T16:41:26.353 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:41:26.353 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 785 k 2024-09-16T16:41:26.353 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 791 k 2024-09-16T16:41:26.353 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-16T16:41:28.736 INFO:teuthology.orchestra.run.smithi083.stdout:cephadm-19.3.0-4879.g81fedcb3.el9.noarch.rpm 329 kB/s | 785 kB 00:02 2024-09-16T16:41:28.736 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:41:28.737 INFO:teuthology.orchestra.run.smithi083.stdout:Total 329 kB/s | 785 kB 00:02 2024-09-16T16:41:28.737 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-16T16:41:28.743 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-16T16:41:28.743 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-16T16:41:28.749 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-16T16:41:28.750 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-16T16:41:28.837 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-16T16:41:29.251 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 1/1 2024-09-16T16:41:29.332 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 1/1 2024-09-16T16:41:29.683 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 1/1 2024-09-16T16:41:29.922 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 1/1 2024-09-16T16:41:29.922 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:41:29.922 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-16T16:41:29.923 INFO:teuthology.orchestra.run.smithi083.stdout: cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T16:41:29.923 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:41:29.923 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-16T16:41:30.197 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T16:41:30.767 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:08 ago on Mon 16 Sep 2024 04:39:22 PM UTC. 2024-09-16T16:41:30.899 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-16T16:41:30.900 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:41:30.900 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-09-16T16:41:30.900 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:41:30.900 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-16T16:41:30.900 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 158 k 2024-09-16T16:41:30.900 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:41:30.900 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-16T16:41:30.900 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:41:30.900 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-09-16T16:41:30.901 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:41:30.901 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 158 k 2024-09-16T16:41:30.901 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 644 k 2024-09-16T16:41:30.901 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-16T16:41:31.601 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-immutable-object-cache-19.3.0-4879.g81fedc 226 kB/s | 158 kB 00:00 2024-09-16T16:41:31.601 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:41:31.602 INFO:teuthology.orchestra.run.smithi083.stdout:Total 225 kB/s | 158 kB 00:00 2024-09-16T16:41:31.602 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-16T16:41:31.612 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-16T16:41:31.612 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-16T16:41:31.671 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-16T16:41:31.671 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-16T16:41:31.813 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-16T16:41:31.918 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4879.g81fedcb3. 1/1 2024-09-16T16:41:31.943 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4879.g81fedcb3. 1/1 2024-09-16T16:41:31.943 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T16:41:31.943 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T16:41:31.943 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:41:32.873 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4879.g81fedcb3. 1/1 2024-09-16T16:41:32.873 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:41:32.873 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-16T16:41:32.873 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:41:32.873 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:41:32.874 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-16T16:41:33.147 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr 2024-09-16T16:41:33.716 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:11 ago on Mon 16 Sep 2024 04:39:22 PM UTC. 2024-09-16T16:41:33.806 INFO:teuthology.orchestra.run.smithi083.stdout:Package ceph-mgr-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T16:41:33.850 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-16T16:41:33.851 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-09-16T16:41:33.851 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-16T16:41:33.939 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T16:41:34.491 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:12 ago on Mon 16 Sep 2024 04:39:22 PM UTC. 2024-09-16T16:41:34.623 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-16T16:41:34.624 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:41:34.624 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-09-16T16:41:34.624 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:41:34.624 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-16T16:41:34.625 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 14 M 2024-09-16T16:41:34.625 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-09-16T16:41:34.625 INFO:teuthology.orchestra.run.smithi083.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T16:41:34.625 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 34 k 2024-09-16T16:41:34.625 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 16 k 2024-09-16T16:41:34.625 INFO:teuthology.orchestra.run.smithi083.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T16:41:34.625 INFO:teuthology.orchestra.run.smithi083.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T16:41:34.625 INFO:teuthology.orchestra.run.smithi083.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T16:41:34.625 INFO:teuthology.orchestra.run.smithi083.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T16:41:34.625 INFO:teuthology.orchestra.run.smithi083.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T16:41:34.625 INFO:teuthology.orchestra.run.smithi083.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T16:41:34.625 INFO:teuthology.orchestra.run.smithi083.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T16:41:34.626 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:41:34.626 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-16T16:41:34.626 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:41:34.626 INFO:teuthology.orchestra.run.smithi083.stdout:Install 11 Packages 2024-09-16T16:41:34.626 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:41:34.626 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 18 M 2024-09-16T16:41:34.626 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 155 M 2024-09-16T16:41:34.627 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-16T16:41:35.380 INFO:teuthology.orchestra.run.smithi083.stdout:(1/11): ceph-grafana-dashboards-19.3.0-4879.g81 77 kB/s | 34 kB 00:00 2024-09-16T16:41:35.405 INFO:teuthology.orchestra.run.smithi083.stdout:(2/11): ceph-prometheus-alerts-19.3.0-4879.g81f 35 kB/s | 16 kB 00:00 2024-09-16T16:41:35.555 INFO:teuthology.orchestra.run.smithi083.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 269 kB 00:00 2024-09-16T16:41:35.580 INFO:teuthology.orchestra.run.smithi083.stdout:(4/11): protobuf-compiler-3.14.0-13.el9.x86_64. 4.8 MB/s | 863 kB 00:00 2024-09-16T16:41:35.639 INFO:teuthology.orchestra.run.smithi083.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 377 kB/s | 19 kB 00:00 2024-09-16T16:41:35.689 INFO:teuthology.orchestra.run.smithi083.stdout:(6/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 4.0 MB/s | 548 kB 00:00 2024-09-16T16:41:35.723 INFO:teuthology.orchestra.run.smithi083.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 4.2 MB/s | 144 kB 00:00 2024-09-16T16:41:35.748 INFO:teuthology.orchestra.run.smithi083.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-16T16:41:35.782 INFO:teuthology.orchestra.run.smithi083.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 5.5 MB/s | 188 kB 00:00 2024-09-16T16:41:35.849 INFO:teuthology.orchestra.run.smithi083.stdout:(10/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 9.7 MB/s | 2.0 MB 00:00 2024-09-16T16:41:59.155 INFO:teuthology.orchestra.run.smithi083.stdout:(11/11): ceph-mgr-dashboard-19.3.0-4879.g81fedc 599 kB/s | 14 MB 00:24 2024-09-16T16:41:59.159 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:41:59.160 INFO:teuthology.orchestra.run.smithi083.stdout:Total 762 kB/s | 18 MB 00:24 2024-09-16T16:41:59.192 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-16T16:41:59.213 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-16T16:41:59.213 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-16T16:41:59.473 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-16T16:41:59.474 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-16T16:42:00.017 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-16T16:42:00.086 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-09-16T16:42:00.117 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-09-16T16:42:00.205 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-09-16T16:42:00.334 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-09-16T16:42:00.395 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-09-16T16:42:00.479 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-09-16T16:42:00.530 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-09-16T16:42:00.559 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-09-16T16:42:00.591 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4879.g81fedcb3.el9 9/11 2024-09-16T16:42:03.775 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4879.g81fedcb3.el 10/11 2024-09-16T16:42:03.846 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4879.g81fedcb3.el9.noa 11/11 2024-09-16T16:42:05.525 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4879.g81fedcb3.el9.noa 11/11 2024-09-16T16:42:05.525 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4879.g81fedcb3.el 1/11 2024-09-16T16:42:05.525 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4879.g81fedcb3.el9.noa 2/11 2024-09-16T16:42:05.525 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4879.g81fedcb3.el9 3/11 2024-09-16T16:42:05.525 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-09-16T16:42:05.525 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-09-16T16:42:05.525 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-09-16T16:42:05.525 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-09-16T16:42:05.526 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-09-16T16:42:05.526 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-09-16T16:42:05.526 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-09-16T16:42:05.845 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-09-16T16:42:05.846 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:05.846 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-16T16:42:05.846 INFO:teuthology.orchestra.run.smithi083.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T16:42:05.846 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-grafana-dashboards-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T16:42:05.846 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-dashboard-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T16:42:05.846 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-prometheus-alerts-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T16:42:05.846 INFO:teuthology.orchestra.run.smithi083.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T16:42:05.846 INFO:teuthology.orchestra.run.smithi083.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T16:42:05.846 INFO:teuthology.orchestra.run.smithi083.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T16:42:05.846 INFO:teuthology.orchestra.run.smithi083.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T16:42:05.846 INFO:teuthology.orchestra.run.smithi083.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T16:42:05.846 INFO:teuthology.orchestra.run.smithi083.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T16:42:05.846 INFO:teuthology.orchestra.run.smithi083.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T16:42:05.847 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:05.847 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-16T16:42:06.162 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T16:42:06.724 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:44 ago on Mon 16 Sep 2024 04:39:22 PM UTC. 2024-09-16T16:42:06.858 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-16T16:42:06.859 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================= 2024-09-16T16:42:06.859 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-09-16T16:42:06.859 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================= 2024-09-16T16:42:06.859 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-16T16:42:06.859 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 7.4 M 2024-09-16T16:42:06.859 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-09-16T16:42:06.859 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T16:42:06.859 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T16:42:06.860 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T16:42:06.860 INFO:teuthology.orchestra.run.smithi083.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T16:42:06.860 INFO:teuthology.orchestra.run.smithi083.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T16:42:06.860 INFO:teuthology.orchestra.run.smithi083.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T16:42:06.860 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T16:42:06.860 INFO:teuthology.orchestra.run.smithi083.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T16:42:06.860 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T16:42:06.860 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T16:42:06.860 INFO:teuthology.orchestra.run.smithi083.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T16:42:06.860 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:06.860 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-16T16:42:06.860 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================= 2024-09-16T16:42:06.861 INFO:teuthology.orchestra.run.smithi083.stdout:Install 12 Packages 2024-09-16T16:42:06.861 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:06.861 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 38 M 2024-09-16T16:42:06.861 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 212 M 2024-09-16T16:42:06.861 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-16T16:42:07.529 INFO:teuthology.orchestra.run.smithi083.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-16T16:42:07.570 INFO:teuthology.orchestra.run.smithi083.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.6 MB/s | 801 kB 00:00 2024-09-16T16:42:07.813 INFO:teuthology.orchestra.run.smithi083.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 115 kB/s | 33 kB 00:00 2024-09-16T16:42:07.938 INFO:teuthology.orchestra.run.smithi083.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 145 kB/s | 18 kB 00:00 2024-09-16T16:42:08.055 INFO:teuthology.orchestra.run.smithi083.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 330 kB/s | 38 kB 00:00 2024-09-16T16:42:08.780 INFO:teuthology.orchestra.run.smithi083.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.5 MB/s | 3.0 MB 00:01 2024-09-16T16:42:08.897 INFO:teuthology.orchestra.run.smithi083.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.0 MB/s | 245 kB 00:00 2024-09-16T16:42:10.748 INFO:teuthology.orchestra.run.smithi083.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 1.8 MB/s | 4.9 MB 00:02 2024-09-16T16:42:11.123 INFO:teuthology.orchestra.run.smithi083.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 669 kB/s | 250 kB 00:00 2024-09-16T16:42:13.706 INFO:teuthology.orchestra.run.smithi083.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 1.1 MB/s | 5.1 MB 00:04 2024-09-16T16:42:21.164 INFO:teuthology.orchestra.run.smithi083.stdout:(11/12): ceph-mgr-diskprediction-local-19.3.0-4 549 kB/s | 7.4 MB 00:13 2024-09-16T16:42:23.922 INFO:teuthology.orchestra.run.smithi083.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 1.3 MB/s | 16 MB 00:12 2024-09-16T16:42:23.923 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:42:23.923 INFO:teuthology.orchestra.run.smithi083.stdout:Total 2.2 MB/s | 38 MB 00:17 2024-09-16T16:42:24.281 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-16T16:42:24.308 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-16T16:42:24.309 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-16T16:42:24.757 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-16T16:42:24.757 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-16T16:42:25.515 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-16T16:42:25.594 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T16:42:25.636 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T16:42:25.736 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T16:42:25.780 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T16:42:25.964 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T16:42:26.012 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T16:42:26.099 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T16:42:26.744 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T16:42:26.859 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T16:42:28.083 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T16:42:29.123 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T16:42:29.162 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4879.g81fed 12/12 2024-09-16T16:42:31.724 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4879.g81fed 12/12 2024-09-16T16:42:31.724 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4879.g81fed 1/12 2024-09-16T16:42:31.724 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T16:42:31.724 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T16:42:31.724 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T16:42:31.724 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T16:42:31.724 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T16:42:31.724 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T16:42:31.724 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T16:42:31.724 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T16:42:31.724 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T16:42:31.725 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T16:42:32.106 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T16:42:32.106 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:32.106 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-16T16:42:32.106 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T16:42:32.106 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T16:42:32.106 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T16:42:32.106 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T16:42:32.106 INFO:teuthology.orchestra.run.smithi083.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T16:42:32.106 INFO:teuthology.orchestra.run.smithi083.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T16:42:32.106 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T16:42:32.106 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T16:42:32.107 INFO:teuthology.orchestra.run.smithi083.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T16:42:32.107 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T16:42:32.107 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T16:42:32.107 INFO:teuthology.orchestra.run.smithi083.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T16:42:32.107 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:32.107 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-16T16:42:32.584 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-rook 2024-09-16T16:42:33.152 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:03:11 ago on Mon 16 Sep 2024 04:39:22 PM UTC. 2024-09-16T16:42:33.292 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-16T16:42:33.293 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:42:33.293 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-09-16T16:42:33.293 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:42:33.293 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-16T16:42:33.293 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-rook noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 49 k 2024-09-16T16:42:33.294 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-09-16T16:42:33.294 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T16:42:33.294 INFO:teuthology.orchestra.run.smithi083.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T16:42:33.294 INFO:teuthology.orchestra.run.smithi083.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T16:42:33.294 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T16:42:33.294 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T16:42:33.294 INFO:teuthology.orchestra.run.smithi083.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T16:42:33.294 INFO:teuthology.orchestra.run.smithi083.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T16:42:33.294 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T16:42:33.294 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T16:42:33.294 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T16:42:33.294 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T16:42:33.294 INFO:teuthology.orchestra.run.smithi083.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T16:42:33.295 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:33.295 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-16T16:42:33.295 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:42:33.295 INFO:teuthology.orchestra.run.smithi083.stdout:Install 13 Packages 2024-09-16T16:42:33.295 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:33.295 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 2.2 M 2024-09-16T16:42:33.295 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 26 M 2024-09-16T16:42:33.296 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-16T16:42:33.713 INFO:teuthology.orchestra.run.smithi083.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 253 kB/s | 19 kB 00:00 2024-09-16T16:42:33.746 INFO:teuthology.orchestra.run.smithi083.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 234 kB/s | 26 kB 00:00 2024-09-16T16:42:33.830 INFO:teuthology.orchestra.run.smithi083.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-16T16:42:33.855 INFO:teuthology.orchestra.run.smithi083.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-16T16:42:33.889 INFO:teuthology.orchestra.run.smithi083.stdout:(5/13): python3-requests-oauthlib-1.3.0-12.el9. 1.6 MB/s | 54 kB 00:00 2024-09-16T16:42:33.922 INFO:teuthology.orchestra.run.smithi083.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.0 MB/s | 279 kB 00:00 2024-09-16T16:42:33.956 INFO:teuthology.orchestra.run.smithi083.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 481 kB/s | 32 kB 00:00 2024-09-16T16:42:33.981 INFO:teuthology.orchestra.run.smithi083.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 241 kB/s | 14 kB 00:00 2024-09-16T16:42:34.006 INFO:teuthology.orchestra.run.smithi083.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 4.4 MB/s | 220 kB 00:00 2024-09-16T16:42:34.023 INFO:teuthology.orchestra.run.smithi083.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 3.5 MB/s | 59 kB 00:00 2024-09-16T16:42:34.049 INFO:teuthology.orchestra.run.smithi083.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.5 MB/s | 90 kB 00:00 2024-09-16T16:42:34.099 INFO:teuthology.orchestra.run.smithi083.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 8.7 MB/s | 1.0 MB 00:00 2024-09-16T16:42:34.174 INFO:teuthology.orchestra.run.smithi083.stdout:(13/13): ceph-mgr-rook-19.3.0-4879.g81fedcb3.el 92 kB/s | 49 kB 00:00 2024-09-16T16:42:34.177 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:42:34.177 INFO:teuthology.orchestra.run.smithi083.stdout:Total 2.5 MB/s | 2.2 MB 00:00 2024-09-16T16:42:34.243 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-16T16:42:34.259 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-16T16:42:34.259 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-16T16:42:34.351 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-16T16:42:34.353 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-16T16:42:34.868 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-16T16:42:35.037 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T16:42:35.092 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T16:42:35.156 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T16:42:35.193 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T16:42:35.235 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T16:42:35.336 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T16:42:35.446 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T16:42:35.503 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T16:42:36.220 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T16:42:36.284 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T16:42:36.317 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T16:42:36.365 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T16:42:36.387 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-rook-2:19.3.0-4879.g81fedcb3.el9.noarch 13/13 2024-09-16T16:42:37.237 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4879.g81fedcb3.el9.noarch 13/13 2024-09-16T16:42:37.237 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4879.g81fedcb3.el9.noarch 1/13 2024-09-16T16:42:37.237 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T16:42:37.237 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T16:42:37.237 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T16:42:37.238 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T16:42:37.238 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T16:42:37.238 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T16:42:37.238 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T16:42:37.238 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T16:42:37.238 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T16:42:37.238 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T16:42:37.238 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T16:42:37.573 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T16:42:37.573 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:37.573 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-16T16:42:37.573 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-rook-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T16:42:37.573 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T16:42:37.574 INFO:teuthology.orchestra.run.smithi083.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T16:42:37.574 INFO:teuthology.orchestra.run.smithi083.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T16:42:37.574 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T16:42:37.574 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T16:42:37.574 INFO:teuthology.orchestra.run.smithi083.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T16:42:37.574 INFO:teuthology.orchestra.run.smithi083.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T16:42:37.574 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T16:42:37.574 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T16:42:37.574 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T16:42:37.574 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T16:42:37.574 INFO:teuthology.orchestra.run.smithi083.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T16:42:37.574 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:37.574 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-16T16:42:37.838 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T16:42:38.404 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:03:16 ago on Mon 16 Sep 2024 04:39:22 PM UTC. 2024-09-16T16:42:38.537 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-16T16:42:38.537 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:42:38.538 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-09-16T16:42:38.538 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:42:38.538 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-16T16:42:38.538 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 154 k 2024-09-16T16:42:38.538 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-09-16T16:42:38.538 INFO:teuthology.orchestra.run.smithi083.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T16:42:38.538 INFO:teuthology.orchestra.run.smithi083.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T16:42:38.538 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T16:42:38.538 INFO:teuthology.orchestra.run.smithi083.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T16:42:38.538 INFO:teuthology.orchestra.run.smithi083.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T16:42:38.538 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:38.538 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-16T16:42:38.538 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:42:38.539 INFO:teuthology.orchestra.run.smithi083.stdout:Install 6 Packages 2024-09-16T16:42:38.539 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:38.539 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 7.0 M 2024-09-16T16:42:38.539 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 33 M 2024-09-16T16:42:38.539 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-16T16:42:38.927 INFO:teuthology.orchestra.run.smithi083.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-09-16T16:42:39.002 INFO:teuthology.orchestra.run.smithi083.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.2 MB/s | 548 kB 00:00 2024-09-16T16:42:39.027 INFO:teuthology.orchestra.run.smithi083.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-16T16:42:39.053 INFO:teuthology.orchestra.run.smithi083.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-16T16:42:39.345 INFO:teuthology.orchestra.run.smithi083.stdout:(5/6): ceph-mgr-cephadm-19.3.0-4879.g81fedcb3.e 250 kB/s | 154 kB 00:00 2024-09-16T16:42:40.061 INFO:teuthology.orchestra.run.smithi083.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.5 MB/s | 6.0 MB 00:01 2024-09-16T16:42:40.063 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:42:40.063 INFO:teuthology.orchestra.run.smithi083.stdout:Total 4.6 MB/s | 7.0 MB 00:01 2024-09-16T16:42:40.134 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-16T16:42:40.145 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-16T16:42:40.145 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-16T16:42:40.245 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-16T16:42:40.246 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-16T16:42:40.551 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-16T16:42:40.724 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T16:42:40.771 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T16:42:41.285 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T16:42:41.367 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T16:42:41.442 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T16:42:41.467 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 6/6 2024-09-16T16:42:42.275 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 6/6 2024-09-16T16:42:42.275 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 1/6 2024-09-16T16:42:42.275 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T16:42:42.275 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T16:42:42.275 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T16:42:42.275 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T16:42:42.548 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T16:42:42.548 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:42.548 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-16T16:42:42.548 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T16:42:42.549 INFO:teuthology.orchestra.run.smithi083.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T16:42:42.549 INFO:teuthology.orchestra.run.smithi083.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T16:42:42.549 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T16:42:42.549 INFO:teuthology.orchestra.run.smithi083.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T16:42:42.549 INFO:teuthology.orchestra.run.smithi083.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T16:42:42.549 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:42.549 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-16T16:42:42.842 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-fuse 2024-09-16T16:42:43.408 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:03:21 ago on Mon 16 Sep 2024 04:39:22 PM UTC. 2024-09-16T16:42:43.541 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-16T16:42:43.541 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:42:43.541 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-09-16T16:42:43.541 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:42:43.541 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-16T16:42:43.542 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 884 k 2024-09-16T16:42:43.542 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-09-16T16:42:43.542 INFO:teuthology.orchestra.run.smithi083.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T16:42:43.542 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:43.542 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-16T16:42:43.542 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:42:43.542 INFO:teuthology.orchestra.run.smithi083.stdout:Install 2 Packages 2024-09-16T16:42:43.542 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:43.543 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 964 k 2024-09-16T16:42:43.543 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 4.1 M 2024-09-16T16:42:43.543 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-16T16:42:44.506 INFO:teuthology.orchestra.run.smithi083.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 112 kB/s | 80 kB 00:00 2024-09-16T16:42:44.705 INFO:teuthology.orchestra.run.smithi083.stdout:(2/2): ceph-fuse-19.3.0-4879.g81fedcb3.el9.x86_ 969 kB/s | 884 kB 00:00 2024-09-16T16:42:44.706 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:42:44.706 INFO:teuthology.orchestra.run.smithi083.stdout:Total 828 kB/s | 964 kB 00:01 2024-09-16T16:42:44.718 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-16T16:42:44.742 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-16T16:42:44.742 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-16T16:42:44.822 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-16T16:42:44.822 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-16T16:42:45.004 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-16T16:42:45.159 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T16:42:45.221 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 2/2 2024-09-16T16:42:45.899 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 2/2 2024-09-16T16:42:45.899 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/2 2024-09-16T16:42:46.163 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T16:42:46.164 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:46.164 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-16T16:42:46.164 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T16:42:46.164 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:46.164 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-16T16:42:46.458 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-volume 2024-09-16T16:42:47.041 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:03:25 ago on Mon 16 Sep 2024 04:39:22 PM UTC. 2024-09-16T16:42:47.176 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-16T16:42:47.176 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:42:47.177 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-09-16T16:42:47.177 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:42:47.177 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-16T16:42:47.177 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-volume noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 285 k 2024-09-16T16:42:47.177 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:47.177 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-16T16:42:47.177 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:42:47.177 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-09-16T16:42:47.177 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:47.177 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 285 k 2024-09-16T16:42:47.177 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 1.3 M 2024-09-16T16:42:47.177 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-16T16:42:47.869 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-volume-19.3.0-4879.g81fedcb3.el9.noarch.rp 412 kB/s | 285 kB 00:00 2024-09-16T16:42:47.870 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:42:47.870 INFO:teuthology.orchestra.run.smithi083.stdout:Total 411 kB/s | 285 kB 00:00 2024-09-16T16:42:47.871 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-16T16:42:47.879 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-16T16:42:47.879 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-16T16:42:47.916 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-16T16:42:47.916 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-16T16:42:48.146 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-16T16:42:48.263 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-volume-2:19.3.0-4879.g81fedcb3.el9.noarch 1/1 2024-09-16T16:42:48.280 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-volume-2:19.3.0-4879.g81fedcb3.el9.noarch 1/1 2024-09-16T16:42:48.280 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T16:42:48.280 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T16:42:48.280 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:49.255 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-volume-2:19.3.0-4879.g81fedcb3.el9.noarch 1/1 2024-09-16T16:42:49.256 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:49.256 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-16T16:42:49.256 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-volume-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T16:42:49.256 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:49.256 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-16T16:42:49.591 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librados-devel 2024-09-16T16:42:50.172 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:03:28 ago on Mon 16 Sep 2024 04:39:22 PM UTC. 2024-09-16T16:42:50.307 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-16T16:42:50.308 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:42:50.308 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-09-16T16:42:50.308 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:42:50.308 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-16T16:42:50.308 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 125 k 2024-09-16T16:42:50.308 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:50.308 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-16T16:42:50.308 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:42:50.308 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-09-16T16:42:50.308 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:50.309 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 125 k 2024-09-16T16:42:50.309 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 445 k 2024-09-16T16:42:50.309 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-16T16:42:50.883 INFO:teuthology.orchestra.run.smithi083.stdout:librados-devel-19.3.0-4879.g81fedcb3.el9.x86_64 217 kB/s | 125 kB 00:00 2024-09-16T16:42:50.884 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:42:50.884 INFO:teuthology.orchestra.run.smithi083.stdout:Total 217 kB/s | 125 kB 00:00 2024-09-16T16:42:50.884 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-16T16:42:50.889 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-16T16:42:50.889 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-16T16:42:50.953 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-16T16:42:50.954 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-16T16:42:51.119 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-16T16:42:51.215 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librados-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:42:51.721 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:42:51.981 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:42:51.982 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:51.982 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-16T16:42:51.982 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:42:51.982 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:51.982 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-16T16:42:52.258 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install libcephfs2 2024-09-16T16:42:52.832 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:03:30 ago on Mon 16 Sep 2024 04:39:22 PM UTC. 2024-09-16T16:42:52.925 INFO:teuthology.orchestra.run.smithi083.stdout:Package libcephfs2-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T16:42:52.972 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-16T16:42:52.973 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-09-16T16:42:52.974 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-16T16:42:53.068 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install libcephfs-devel 2024-09-16T16:42:53.632 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:03:31 ago on Mon 16 Sep 2024 04:39:22 PM UTC. 2024-09-16T16:42:53.765 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-16T16:42:53.766 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:42:53.766 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-09-16T16:42:53.766 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:42:53.766 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-16T16:42:53.766 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs-devel x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 32 k 2024-09-16T16:42:53.766 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:53.766 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-16T16:42:53.766 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:42:53.767 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-09-16T16:42:53.767 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:53.767 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 32 k 2024-09-16T16:42:53.767 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 142 k 2024-09-16T16:42:53.767 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-16T16:42:54.187 INFO:teuthology.orchestra.run.smithi083.stdout:libcephfs-devel-19.3.0-4879.g81fedcb3.el9.x86_6 75 kB/s | 32 kB 00:00 2024-09-16T16:42:54.188 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:42:54.188 INFO:teuthology.orchestra.run.smithi083.stdout:Total 75 kB/s | 32 kB 00:00 2024-09-16T16:42:54.188 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-16T16:42:54.191 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-16T16:42:54.192 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-16T16:42:54.203 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-16T16:42:54.203 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-16T16:42:54.294 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-16T16:42:54.410 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephfs-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:42:54.767 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:42:55.011 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephfs-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:42:55.011 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:55.011 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-16T16:42:55.012 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:42:55.012 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:42:55.012 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-16T16:42:55.227 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librados2 2024-09-16T16:42:55.789 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:03:33 ago on Mon 16 Sep 2024 04:39:22 PM UTC. 2024-09-16T16:42:55.879 INFO:teuthology.orchestra.run.smithi083.stdout:Package librados2-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T16:42:55.922 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-16T16:42:55.923 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-09-16T16:42:55.923 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-16T16:42:56.020 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librbd1 2024-09-16T16:42:56.585 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:03:34 ago on Mon 16 Sep 2024 04:39:22 PM UTC. 2024-09-16T16:42:56.675 INFO:teuthology.orchestra.run.smithi083.stdout:Package librbd1-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T16:42:56.717 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-16T16:42:56.719 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-09-16T16:42:56.719 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-16T16:42:56.810 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rados 2024-09-16T16:42:57.376 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:03:35 ago on Mon 16 Sep 2024 04:39:22 PM UTC. 2024-09-16T16:42:57.466 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-rados-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T16:42:57.509 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-16T16:42:57.511 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-09-16T16:42:57.511 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-16T16:42:57.611 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rgw 2024-09-16T16:42:58.172 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:03:36 ago on Mon 16 Sep 2024 04:39:22 PM UTC. 2024-09-16T16:42:58.263 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-rgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T16:42:58.306 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-16T16:42:58.307 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-09-16T16:42:58.308 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-16T16:42:58.394 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-cephfs 2024-09-16T16:42:58.960 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:03:36 ago on Mon 16 Sep 2024 04:39:22 PM UTC. 2024-09-16T16:42:59.051 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-cephfs-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T16:42:59.094 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-16T16:42:59.096 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-09-16T16:42:59.096 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-16T16:42:59.186 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rbd 2024-09-16T16:42:59.748 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:03:37 ago on Mon 16 Sep 2024 04:39:22 PM UTC. 2024-09-16T16:42:59.839 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-rbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T16:42:59.882 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-16T16:42:59.883 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-09-16T16:42:59.883 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-16T16:42:59.977 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-fuse 2024-09-16T16:43:00.537 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:03:38 ago on Mon 16 Sep 2024 04:39:22 PM UTC. 2024-09-16T16:43:00.670 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-16T16:43:00.671 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:43:00.671 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-09-16T16:43:00.671 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:43:00.671 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-16T16:43:00.671 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 91 k 2024-09-16T16:43:00.671 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:00.671 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-16T16:43:00.671 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:43:00.672 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-09-16T16:43:00.672 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:00.672 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 91 k 2024-09-16T16:43:00.672 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 321 k 2024-09-16T16:43:00.672 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-16T16:43:01.258 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-fuse-19.3.0-4879.g81fedcb3.el9.x86_64.rpm 155 kB/s | 91 kB 00:00 2024-09-16T16:43:01.258 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:43:01.259 INFO:teuthology.orchestra.run.smithi083.stdout:Total 155 kB/s | 91 kB 00:00 2024-09-16T16:43:01.259 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-16T16:43:01.267 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-16T16:43:01.268 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-16T16:43:01.331 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-16T16:43:01.331 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-16T16:43:01.479 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-16T16:43:01.580 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:43:01.973 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:43:02.220 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:43:02.220 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:02.221 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-16T16:43:02.221 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:43:02.221 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:02.221 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-16T16:43:02.514 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-mirror 2024-09-16T16:43:03.088 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:03:41 ago on Mon 16 Sep 2024 04:39:22 PM UTC. 2024-09-16T16:43:03.225 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-16T16:43:03.226 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:43:03.226 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-09-16T16:43:03.226 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:43:03.226 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-16T16:43:03.226 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 3.5 M 2024-09-16T16:43:03.226 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:03.226 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-16T16:43:03.226 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:43:03.226 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-09-16T16:43:03.226 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:03.227 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 3.5 M 2024-09-16T16:43:03.227 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 30 M 2024-09-16T16:43:03.227 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-16T16:43:06.337 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-mirror-19.3.0-4879.g81fedcb3.el9.x86_64.rpm 1.1 MB/s | 3.5 MB 00:03 2024-09-16T16:43:06.338 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:43:06.339 INFO:teuthology.orchestra.run.smithi083.stdout:Total 1.1 MB/s | 3.5 MB 00:03 2024-09-16T16:43:06.339 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-16T16:43:06.352 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-16T16:43:06.352 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-16T16:43:06.439 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-16T16:43:06.439 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-16T16:43:07.042 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-16T16:43:07.113 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-mirror-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:43:07.138 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:43:07.139 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T16:43:07.139 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T16:43:07.139 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T16:43:07.139 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T16:43:07.139 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:08.405 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-mirror-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:43:08.405 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:08.406 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-16T16:43:08.406 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:43:08.406 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:08.406 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-16T16:43:08.678 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-nbd 2024-09-16T16:43:09.263 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:03:47 ago on Mon 16 Sep 2024 04:39:22 PM UTC. 2024-09-16T16:43:09.399 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-16T16:43:09.399 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:43:09.399 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-09-16T16:43:09.399 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:43:09.400 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-16T16:43:09.400 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 177 k 2024-09-16T16:43:09.400 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:09.400 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-16T16:43:09.400 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-16T16:43:09.400 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-09-16T16:43:09.400 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:09.401 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 177 k 2024-09-16T16:43:09.401 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 685 k 2024-09-16T16:43:09.401 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-16T16:43:10.067 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-nbd-19.3.0-4879.g81fedcb3.el9.x86_64.rpm 265 kB/s | 177 kB 00:00 2024-09-16T16:43:10.068 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:43:10.068 INFO:teuthology.orchestra.run.smithi083.stdout:Total 265 kB/s | 177 kB 00:00 2024-09-16T16:43:10.068 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-16T16:43:10.078 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-16T16:43:10.078 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-16T16:43:10.149 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-16T16:43:10.150 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-16T16:43:10.328 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-16T16:43:10.423 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-nbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:43:10.865 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:43:11.113 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-nbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:43:11.113 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:11.114 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-16T16:43:11.114 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:43:11.114 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:11.114 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-16T16:43:11.404 DEBUG:teuthology.parallel:result is None 2024-09-16T16:43:11.405 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=81fedcb3b484c843d22ffbc8d179cb567b7083b5 2024-09-16T16:43:11.539 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T16:43:11.561 INFO:teuthology.orchestra.run.smithi083.stdout:19.3.0-4879.g81fedcb3.el9 2024-09-16T16:43:11.561 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4879.g81fedcb3.el9 2024-09-16T16:43:11.561 INFO:teuthology.task.install:The correct ceph version 19.3.0-4879.g81fedcb3 is installed. 2024-09-16T16:43:11.563 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T16:43:11.563 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T16:43:11.563 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T16:43:11.627 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T16:43:11.627 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T16:43:11.627 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T16:43:11.692 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T16:43:11.756 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T16:43:11.756 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T16:43:11.756 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T16:43:11.820 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T16:43:11.884 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T16:43:11.885 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T16:43:11.885 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T16:43:11.949 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T16:43:12.014 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T16:43:12.100 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T16:43:12.100 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 777 /var/log/ceph 2024-09-16T16:43:12.124 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T16:43:12.124 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T16:43:12.219 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T16:43:12.219 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T16:43:12.250 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-16T16:43:12.250 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr stats period': 30}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T16:43:12.250 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2024-09-16_16:23:48-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi/7907376', 'branch': 'wip-matanb-crimson-testing-59543', 'description': 'crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_lock_and_fence}', 'email': None, 'first_in_suite': False, 'job_id': '7907376', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2024-09-16_16:23:48-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-matanb-crimson-testing-59543'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr stats period': 30}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '81fedcb3b484c843d22ffbc8d179cb567b7083b5'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '81fedcb3b484c843d22ffbc8d179cb567b7083b5'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '0325fd3c11a225c0fd9eb38fab88aad1370ad779'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 6678, 'sha1': '81fedcb3b484c843d22ffbc8d179cb567b7083b5', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '0325fd3c11a225c0fd9eb38fab88aad1370ad779', 'targets': {'smithi083.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOd2VVGJeFxu1bFV1Fg98tvaRHY33+WNcqLgarxVlS+AjgSryXYKMVYMrY7vKKFz3hN/KIp6o2da0Rylkz9tLfk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr stats period': 30}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '81fedcb3b484c843d22ffbc8d179cb567b7083b5', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['rbd/test_lock_fence.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-16_16:23:48', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T16:43:12.251 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T16:43:12.299 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T16:43:12.363 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T16:43:12.363 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T16:43:12.419 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T16:43:12.420 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_1 2024-09-16T16:43:12.475 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T16:43:12.475 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T16:43:12.475 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-16T16:43:12.475 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T16:43:12.475 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T16:43:12.475 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-09-16 16:43:07.413774907 +0000 2024-09-16T16:43:12.475 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-09-16 16:38:34.262475103 +0000 2024-09-16T16:43:12.475 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-09-16 16:38:34.262475103 +0000 2024-09-16T16:43:12.476 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-09-16 16:38:34.262475103 +0000 2024-09-16T16:43:12.476 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T16:43:12.539 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-09-16T16:43:12.539 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-09-16T16:43:12.539 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000200961 s, 2.5 MB/s 2024-09-16T16:43:12.540 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T16:43:12.597 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_2 2024-09-16T16:43:12.652 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T16:43:12.652 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T16:43:12.652 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-16T16:43:12.652 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T16:43:12.653 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T16:43:12.653 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-09-16 16:43:07.413774907 +0000 2024-09-16T16:43:12.653 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-09-16 16:38:34.660465709 +0000 2024-09-16T16:43:12.653 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-09-16 16:38:34.660465709 +0000 2024-09-16T16:43:12.653 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-09-16 16:38:34.660465709 +0000 2024-09-16T16:43:12.653 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T16:43:12.715 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-09-16T16:43:12.715 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-09-16T16:43:12.716 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.00022899 s, 2.2 MB/s 2024-09-16T16:43:12.717 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T16:43:12.773 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_3 2024-09-16T16:43:12.828 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T16:43:12.828 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T16:43:12.828 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-16T16:43:12.828 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T16:43:12.828 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T16:43:12.828 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-09-16 16:43:07.414774884 +0000 2024-09-16T16:43:12.828 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-09-16 16:38:35.041456716 +0000 2024-09-16T16:43:12.828 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-09-16 16:38:35.041456716 +0000 2024-09-16T16:43:12.828 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-09-16 16:38:35.041456716 +0000 2024-09-16T16:43:12.828 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T16:43:12.891 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-09-16T16:43:12.891 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-09-16T16:43:12.891 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000206606 s, 2.5 MB/s 2024-09-16T16:43:12.892 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T16:43:12.948 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_4 2024-09-16T16:43:13.003 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T16:43:13.003 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T16:43:13.003 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-16T16:43:13.003 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T16:43:13.003 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T16:43:13.003 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-09-16 16:43:07.414774884 +0000 2024-09-16T16:43:13.003 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-09-16 16:38:35.413447936 +0000 2024-09-16T16:43:13.003 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-09-16 16:38:35.413447936 +0000 2024-09-16T16:43:13.003 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-09-16 16:38:35.413447936 +0000 2024-09-16T16:43:13.004 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T16:43:13.066 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-09-16T16:43:13.066 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-09-16T16:43:13.066 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000164019 s, 3.1 MB/s 2024-09-16T16:43:13.067 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T16:43:13.123 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T16:43:13.123 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi083.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-16T16:43:13.123 INFO:tasks.ceph:Generating config... 2024-09-16T16:43:13.124 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-09-16T16:43:13.124 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-16T16:43:13.124 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-16T16:43:13.124 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-16T16:43:13.124 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-09-16T16:43:13.124 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T16:43:13.124 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T16:43:13.124 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T16:43:13.125 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-09-16T16:43:13.125 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-16T16:43:13.125 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-09-16T16:43:13.125 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-09-16T16:43:13.125 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T16:43:13.125 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T16:43:13.125 INFO:tasks.ceph:[osd] osd objectstore = seastore 2024-09-16T16:43:13.125 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-16T16:43:13.125 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-09-16T16:43:13.125 INFO:tasks.ceph:[global] ms client mode = crc 2024-09-16T16:43:13.125 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-09-16T16:43:13.125 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-09-16T16:43:13.125 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-09-16T16:43:13.125 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-09-16T16:43:13.126 INFO:tasks.ceph:[global] ms service mode = crc 2024-09-16T16:43:13.126 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T16:43:13.126 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T16:43:13.126 INFO:tasks.ceph:[mgr] mgr stats period = 30 2024-09-16T16:43:13.126 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-1 2024-09-16T16:43:13.126 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 2-3 2024-09-16T16:43:13.126 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 4-5 2024-09-16T16:43:13.126 INFO:tasks.ceph:Setting up mon.a... 2024-09-16T16:43:13.126 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-16T16:43:13.217 INFO:teuthology.orchestra.run.smithi083.stdout:creating /etc/ceph/ceph.keyring 2024-09-16T16:43:13.220 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-16T16:43:13.271 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T16:43:13.295 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.83')] 2024-09-16T16:43:13.296 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.83', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'mgr stats period': 30}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}} 2024-09-16T16:43:13.296 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T16:43:13.296 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T16:43:13.351 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.83 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-16T16:43:13.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:13.442+0000 7f25f743ff00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:13.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:13.442+0000 7f25f743ff00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:13.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:13.442+0000 7f25f743ff00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:13.443 INFO:teuthology.orchestra.run.smithi083.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-16T16:43:13.443 INFO:teuthology.orchestra.run.smithi083.stdout:monmaptool: generated fsid 9917636f-e891-42c5-96cb-136e608bdb04 2024-09-16T16:43:13.443 INFO:teuthology.orchestra.run.smithi083.stdout:setting min_mon_release = quincy 2024-09-16T16:43:13.443 INFO:teuthology.orchestra.run.smithi083.stdout:epoch 0 2024-09-16T16:43:13.444 INFO:teuthology.orchestra.run.smithi083.stdout:fsid 9917636f-e891-42c5-96cb-136e608bdb04 2024-09-16T16:43:13.444 INFO:teuthology.orchestra.run.smithi083.stdout:last_changed 2024-09-16T16:43:13.443966+0000 2024-09-16T16:43:13.444 INFO:teuthology.orchestra.run.smithi083.stdout:created 2024-09-16T16:43:13.443966+0000 2024-09-16T16:43:13.444 INFO:teuthology.orchestra.run.smithi083.stdout:min_mon_release 17 (quincy) 2024-09-16T16:43:13.444 INFO:teuthology.orchestra.run.smithi083.stdout:election_strategy: 1 2024-09-16T16:43:13.444 INFO:teuthology.orchestra.run.smithi083.stdout:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-09-16T16:43:13.444 INFO:teuthology.orchestra.run.smithi083.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-09-16T16:43:13.445 DEBUG:teuthology.orchestra.run.smithi083:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T16:43:13.460 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 9917636f-e891-42c5-96cb-136e608bdb04... 2024-09-16T16:43:13.461 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T16:43:13.542 INFO:teuthology.orchestra.run.smithi083.stdout:[global] 2024-09-16T16:43:13.542 INFO:teuthology.orchestra.run.smithi083.stdout: chdir = "" 2024-09-16T16:43:13.542 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T16:43:13.543 INFO:teuthology.orchestra.run.smithi083.stdout: auth supported = cephx 2024-09-16T16:43:13.543 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:13.543 INFO:teuthology.orchestra.run.smithi083.stdout: filestore xattr use omap = true 2024-09-16T16:43:13.543 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:13.543 INFO:teuthology.orchestra.run.smithi083.stdout: mon clock drift allowed = 1.000 2024-09-16T16:43:13.543 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:13.543 INFO:teuthology.orchestra.run.smithi083.stdout: osd crush chooseleaf type = 0 2024-09-16T16:43:13.543 INFO:teuthology.orchestra.run.smithi083.stdout: auth debug = true 2024-09-16T16:43:13.543 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:13.543 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on old message = true 2024-09-16T16:43:13.543 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on bug = true 2024-09-16T16:43:13.543 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:13.543 INFO:teuthology.orchestra.run.smithi083.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T16:43:13.544 INFO:teuthology.orchestra.run.smithi083.stdout: mon pg warn max object skew = 0 2024-09-16T16:43:13.544 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:13.544 INFO:teuthology.orchestra.run.smithi083.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T16:43:13.544 INFO:teuthology.orchestra.run.smithi083.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T16:43:13.544 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:13.544 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default size = 2 2024-09-16T16:43:13.544 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:13.544 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow primary affinity = true 2024-09-16T16:43:13.544 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow pg remap = true 2024-09-16T16:43:13.544 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on legacy crush tunables = false 2024-09-16T16:43:13.544 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on crush straw calc version zero = false 2024-09-16T16:43:13.544 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on no sortbitwise = false 2024-09-16T16:43:13.544 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on osd down out interval zero = false 2024-09-16T16:43:13.545 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on too few osds = false 2024-09-16T16:43:13.545 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T16:43:13.545 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T16:43:13.545 INFO:teuthology.orchestra.run.smithi083.stdout: mon_allow_pool_size_one = true 2024-09-16T16:43:13.545 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:13.545 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T16:43:13.545 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:13.545 INFO:teuthology.orchestra.run.smithi083.stdout: osd default data pool replay window = 5 2024-09-16T16:43:13.545 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:13.545 INFO:teuthology.orchestra.run.smithi083.stdout: mon allow pool delete = true 2024-09-16T16:43:13.545 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:13.545 INFO:teuthology.orchestra.run.smithi083.stdout: mon cluster log file level = debug 2024-09-16T16:43:13.545 INFO:teuthology.orchestra.run.smithi083.stdout: debug asserts on shutdown = true 2024-09-16T16:43:13.546 INFO:teuthology.orchestra.run.smithi083.stdout: mon health detail to clog = false 2024-09-16T16:43:13.546 INFO:teuthology.orchestra.run.smithi083.stdout: mon host = 172.21.15.83 2024-09-16T16:43:13.546 INFO:teuthology.orchestra.run.smithi083.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-09-16T16:43:13.546 INFO:teuthology.orchestra.run.smithi083.stdout: ms client mode = crc 2024-09-16T16:43:13.546 INFO:teuthology.orchestra.run.smithi083.stdout: ms cluster mode = crc 2024-09-16T16:43:13.546 INFO:teuthology.orchestra.run.smithi083.stdout: ms mon client mode = crc 2024-09-16T16:43:13.546 INFO:teuthology.orchestra.run.smithi083.stdout: ms mon cluster mode = crc 2024-09-16T16:43:13.546 INFO:teuthology.orchestra.run.smithi083.stdout: ms mon service mode = crc 2024-09-16T16:43:13.546 INFO:teuthology.orchestra.run.smithi083.stdout: ms service mode = crc 2024-09-16T16:43:13.546 INFO:teuthology.orchestra.run.smithi083.stdout: fsid = 9917636f-e891-42c5-96cb-136e608bdb04 2024-09-16T16:43:13.546 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:13.546 INFO:teuthology.orchestra.run.smithi083.stdout:[osd] 2024-09-16T16:43:13.546 INFO:teuthology.orchestra.run.smithi083.stdout: osd journal size = 100 2024-09-16T16:43:13.546 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:13.547 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub load threshold = 5.0 2024-09-16T16:43:13.547 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub max interval = 600 2024-09-16T16:43:13.547 INFO:teuthology.orchestra.run.smithi083.stdout: osd mclock profile = high_recovery_ops 2024-09-16T16:43:13.547 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:13.547 INFO:teuthology.orchestra.run.smithi083.stdout: osd recover clone overlap = true 2024-09-16T16:43:13.547 INFO:teuthology.orchestra.run.smithi083.stdout: osd recovery max chunk = 1048576 2024-09-16T16:43:13.547 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:13.547 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug shutdown = true 2024-09-16T16:43:13.547 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug op order = true 2024-09-16T16:43:13.547 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug verify stray on activate = true 2024-09-16T16:43:13.547 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug trim objects = true 2024-09-16T16:43:13.547 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:13.547 INFO:teuthology.orchestra.run.smithi083.stdout: osd open classes on start = true 2024-09-16T16:43:13.547 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug pg log writeout = true 2024-09-16T16:43:13.547 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:13.548 INFO:teuthology.orchestra.run.smithi083.stdout: osd deep scrub update digest min age = 30 2024-09-16T16:43:13.548 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:13.548 INFO:teuthology.orchestra.run.smithi083.stdout: osd map max advance = 10 2024-09-16T16:43:13.548 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:13.548 INFO:teuthology.orchestra.run.smithi083.stdout: journal zero on create = true 2024-09-16T16:43:13.548 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:13.548 INFO:teuthology.orchestra.run.smithi083.stdout: filestore ondisk finisher threads = 3 2024-09-16T16:43:13.548 INFO:teuthology.orchestra.run.smithi083.stdout: filestore apply finisher threads = 3 2024-09-16T16:43:13.548 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:13.548 INFO:teuthology.orchestra.run.smithi083.stdout: bdev debug aio = true 2024-09-16T16:43:13.548 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug misdirected ops = true 2024-09-16T16:43:13.548 INFO:teuthology.orchestra.run.smithi083.stdout: debug monc = 20 2024-09-16T16:43:13.548 INFO:teuthology.orchestra.run.smithi083.stdout: crimson alien thread cpu cores = 6-7 2024-09-16T16:43:13.549 INFO:teuthology.orchestra.run.smithi083.stdout: crimson osd obc lru size = 10 2024-09-16T16:43:13.549 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-09-16T16:43:13.549 INFO:teuthology.orchestra.run.smithi083.stdout: debug osd = 20 2024-09-16T16:43:13.549 INFO:teuthology.orchestra.run.smithi083.stdout: osd objectstore = seastore 2024-09-16T16:43:13.549 INFO:teuthology.orchestra.run.smithi083.stdout: osd shutdown pgref assert = True 2024-09-16T16:43:13.549 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:13.549 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr] 2024-09-16T16:43:13.549 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-09-16T16:43:13.549 INFO:teuthology.orchestra.run.smithi083.stdout: debug mgr = 20 2024-09-16T16:43:13.549 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2024-09-16T16:43:13.549 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-09-16T16:43:13.549 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-09-16T16:43:13.549 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-09-16T16:43:13.549 INFO:teuthology.orchestra.run.smithi083.stdout: mgr/telemetry/nag = false 2024-09-16T16:43:13.550 INFO:teuthology.orchestra.run.smithi083.stdout: mgr stats period = 30 2024-09-16T16:43:13.550 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:13.550 INFO:teuthology.orchestra.run.smithi083.stdout:[mon] 2024-09-16T16:43:13.550 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-09-16T16:43:13.550 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2024-09-16T16:43:13.550 INFO:teuthology.orchestra.run.smithi083.stdout: debug paxos = 20 2024-09-16T16:43:13.550 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-09-16T16:43:13.550 INFO:teuthology.orchestra.run.smithi083.stdout: mon data avail warn = 5 2024-09-16T16:43:13.550 INFO:teuthology.orchestra.run.smithi083.stdout: mon mgr mkfs grace = 240 2024-09-16T16:43:13.550 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-09-16T16:43:13.550 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd reporter subtree level = osd 2024-09-16T16:43:13.550 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd prime pg temp = true 2024-09-16T16:43:13.550 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-09-16T16:43:13.550 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:13.551 INFO:teuthology.orchestra.run.smithi083.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T16:43:13.551 INFO:teuthology.orchestra.run.smithi083.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T16:43:13.551 INFO:teuthology.orchestra.run.smithi083.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T16:43:13.551 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:13.551 INFO:teuthology.orchestra.run.smithi083.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T16:43:13.551 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T16:43:13.551 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T16:43:13.551 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:13.551 INFO:teuthology.orchestra.run.smithi083.stdout: # 1m isn't quite enough 2024-09-16T16:43:13.551 INFO:teuthology.orchestra.run.smithi083.stdout: mon_down_mkfs_grace = 2m 2024-09-16T16:43:13.551 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:13.551 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_filestore_osds = false 2024-09-16T16:43:13.551 INFO:teuthology.orchestra.run.smithi083.stdout: mon min osdmap epochs = 50 2024-09-16T16:43:13.551 INFO:teuthology.orchestra.run.smithi083.stdout: mon osdmap full prune interval = 2 2024-09-16T16:43:13.552 INFO:teuthology.orchestra.run.smithi083.stdout: mon osdmap full prune min = 15 2024-09-16T16:43:13.552 INFO:teuthology.orchestra.run.smithi083.stdout: mon osdmap full prune txsize = 2 2024-09-16T16:43:13.552 INFO:teuthology.orchestra.run.smithi083.stdout: paxos service trim min = 10 2024-09-16T16:43:13.552 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default crimson = True 2024-09-16T16:43:13.552 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:13.552 INFO:teuthology.orchestra.run.smithi083.stdout:[client] 2024-09-16T16:43:13.552 INFO:teuthology.orchestra.run.smithi083.stdout: rgw cache enabled = true 2024-09-16T16:43:13.552 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable ops log = true 2024-09-16T16:43:13.552 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable usage log = true 2024-09-16T16:43:13.552 INFO:teuthology.orchestra.run.smithi083.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T16:43:13.552 INFO:teuthology.orchestra.run.smithi083.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T16:43:13.552 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.a] 2024-09-16T16:43:13.552 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.0] 2024-09-16T16:43:13.552 INFO:teuthology.orchestra.run.smithi083.stdout: crimson seastar cpu cores = 0-1 2024-09-16T16:43:13.553 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.1] 2024-09-16T16:43:13.553 INFO:teuthology.orchestra.run.smithi083.stdout: crimson seastar cpu cores = 2-3 2024-09-16T16:43:13.553 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.2] 2024-09-16T16:43:13.553 INFO:teuthology.orchestra.run.smithi083.stdout: crimson seastar cpu cores = 4-5 2024-09-16T16:43:13.554 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-16T16:43:13.554 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-16T16:43:13.659 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T16:43:13.659 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T16:43:13.660 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-16T16:43:13.675 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T16:43:13.675 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-16T16:43:13.730 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi083.front.sepia.ceph.com 2024-09-16T16:43:13.730 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T16:43:13.730 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T16:43:13.730 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T16:43:13.804 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T16:43:13.804 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T16:43:13.859 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T16:43:13.860 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T16:43:13.860 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T16:43:13.959 INFO:teuthology.orchestra.run.smithi083.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T16:43:13.962 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T16:43:13.962 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T16:43:13.962 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T16:43:14.011 INFO:teuthology.orchestra.run.smithi083.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-16T16:43:14.025 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T16:43:14.025 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi083.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-16T16:43:14.025 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-16T16:43:14.049 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T16:43:14.049 INFO:tasks.ceph:role: osd.0 2024-09-16T16:43:14.050 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi083.front.sepia.ceph.com 2024-09-16T16:43:14.050 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T16:43:14.122 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T16:43:14.122 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T16:43:14.122 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T16:43:14.122 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T16:43:14.123 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T16:43:14.123 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2024-09-16T16:43:14.123 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T16:43:14.123 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T16:43:14.123 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T16:43:14.123 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T16:43:14.631 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2024-09-16T16:43:14.633 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2024-09-16T16:43:14.633 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-16T16:43:14.774 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-16T16:43:14.806 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-16T16:43:14.871 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T16:43:14.871 INFO:tasks.ceph:role: osd.1 2024-09-16T16:43:14.872 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi083.front.sepia.ceph.com 2024-09-16T16:43:14.872 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T16:43:14.944 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T16:43:14.944 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T16:43:14.944 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T16:43:14.944 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T16:43:14.944 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T16:43:14.944 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2024-09-16T16:43:14.944 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T16:43:14.944 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T16:43:14.944 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T16:43:14.944 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T16:43:15.397 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2024-09-16T16:43:15.399 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2024-09-16T16:43:15.399 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-16T16:43:15.492 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-16T16:43:15.521 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-16T16:43:15.585 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T16:43:15.586 INFO:tasks.ceph:role: osd.2 2024-09-16T16:43:15.586 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi083.front.sepia.ceph.com 2024-09-16T16:43:15.586 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T16:43:15.659 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T16:43:15.659 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T16:43:15.659 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T16:43:15.659 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T16:43:15.659 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T16:43:15.659 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2024-09-16T16:43:15.659 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T16:43:15.659 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T16:43:15.659 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T16:43:15.660 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T16:43:16.141 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2024-09-16T16:43:16.143 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2024-09-16T16:43:16.143 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-16T16:43:16.293 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-16T16:43:16.322 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T16:43:16.619 INFO:teuthology.orchestra.run.smithi083.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-16T16:43:16.620 INFO:teuthology.orchestra.run.smithi083.stderr:INFO 2024-09-16 16:43:16,620 seastar - Reactor backend: linux-aio 2024-09-16T16:43:16.633 INFO:teuthology.orchestra.run.smithi083.stderr:==28122==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-16T16:43:16.749 INFO:teuthology.orchestra.run.smithi083.stderr:INFO 2024-09-16 16:43:16,750 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-16T16:43:16.749 INFO:teuthology.orchestra.run.smithi083.stderr:INFO 2024-09-16 16:43:16,750 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 0-1 2024-09-16T16:43:16.750 INFO:teuthology.orchestra.run.smithi083.stderr:DEBUG 2024-09-16 16:43:16,751 [shard 0:main] seastar - reactor::drain 2024-09-16T16:43:17.280 INFO:teuthology.orchestra.run.smithi083.stderr:----------------------------------------------------- 2024-09-16T16:43:17.280 INFO:teuthology.orchestra.run.smithi083.stderr:Suppressions used: 2024-09-16T16:43:17.280 INFO:teuthology.orchestra.run.smithi083.stderr: count bytes template 2024-09-16T16:43:17.281 INFO:teuthology.orchestra.run.smithi083.stderr: 1 8 MallocExtension::Register 2024-09-16T16:43:17.281 INFO:teuthology.orchestra.run.smithi083.stderr:----------------------------------------------------- 2024-09-16T16:43:17.281 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-16T16:43:17.335 INFO:teuthology.orchestra.run.smithi083.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-16T16:43:17.338 INFO:teuthology.orchestra.run.smithi083.stderr:INFO 2024-09-16 16:43:17,339 seastar - Reactor backend: linux-aio 2024-09-16T16:43:17.346 INFO:teuthology.orchestra.run.smithi083.stderr:==28120==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-16T16:43:17.628 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x133c44 0xdd332 0x4b7ec 0xb9ea3fb 0xb9ea7b4 0xb9ea95c 0xb9ec717 0xb9ec913 0xba317fd 0xb9d8ccf 0xb9dc6f8 0xb9c7696 0xb9c7a7a 0x9afa023 0x9b11b72 0x99a2168 0x9942951 0x927469d 0x932dfed 0x932ef8c 0x932f00f 0x932f094 0x932f196 0x932f28e 0x932f314 0x932f3cd 0x932f583 0x932f6bb 0x932f736 0x932f7bb 0x932f916 0x932fa30 0x932fab5 0x932fb93 0x9552886 0x961b5f3 0x961b770 0x96dc4d1 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T16:43:17.628 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:17.849 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x10798d 0xdd2fc 0x4ab1c 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T16:43:17.849 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:17.906 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x4ab5b 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T16:43:17.907 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:17.921 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x5542761 0x9e3397b 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T16:43:17.921 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:17.979 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x4ab07 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T16:43:17.979 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:18.014 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x4ac00 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T16:43:18.014 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:18.086 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 230 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x55426b7 0x9e3397b 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T16:43:18.086 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:18.386 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x42198 0x3ecea 0xfc9b1 0x386a219 0x98436b4 0x9af9f5a 0x9b11b72 0x99a2168 0x9942951 0x927469d 0x9311f16 0x93129ba 0x9312a3d 0x9312ac2 0x9312bc4 0x9312cbc 0x9312d42 0x9312dfb 0x9312fb1 0x93130e9 0x9313164 0x93131e9 0x9313344 0x931345e 0x93134e3 0x93135c1 0x9552886 0x961b5f3 0x961b770 0x96dc4d1 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T16:43:18.386 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff8ba9080a 0xffffffff8bda8239 0xffffffff8bd8e2a3 0xffffffff8bd96a3b 0xffffffff8bd96ebc 0xffffffff8ba8c884 0xffffffff8c6d1d62 0xffffffff8c800bb2 2024-09-16T16:43:18.437 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 32 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x4abc8 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T16:43:18.437 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:18.495 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 90 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x9e3395d 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T16:43:18.495 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:19.516 INFO:teuthology.orchestra.run.smithi083.stderr:----------------------------------------------------- 2024-09-16T16:43:19.516 INFO:teuthology.orchestra.run.smithi083.stderr:Suppressions used: 2024-09-16T16:43:19.516 INFO:teuthology.orchestra.run.smithi083.stderr: count bytes template 2024-09-16T16:43:19.516 INFO:teuthology.orchestra.run.smithi083.stderr: 1 8 MallocExtension::Register 2024-09-16T16:43:19.516 INFO:teuthology.orchestra.run.smithi083.stderr:----------------------------------------------------- 2024-09-16T16:43:19.516 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-16T16:43:19.541 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-16T16:43:19.565 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T16:43:19.839 INFO:teuthology.orchestra.run.smithi083.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-16T16:43:19.840 INFO:teuthology.orchestra.run.smithi083.stderr:INFO 2024-09-16 16:43:19,840 seastar - Reactor backend: linux-aio 2024-09-16T16:43:19.845 INFO:teuthology.orchestra.run.smithi083.stderr:==28167==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-16T16:43:19.952 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x10d76c 0xdd31a 0x4b14c 0x381b234 0x5fdd423 0x5fdd8fa 0x6038b24 0x5fb8d06 0x5fb8ee0 0x5fb946c 0x5e00fef 0x5e19231 0x71579d2 0x71691a5 0x716a642 0x716a6c7 0x716a7c9 0x716a8c1 0x716aace 0x716acb4 0x716b003 0x716b3a7 0x716b5d7 0x716b841 0x716bb0e 0x716bd2c 0x716bde4 0x716beaa 0x716bf8b 0x5c63e30 0x5c63f3a 0x5c63ff6 0x5c640b2 0x5c6416e 0x5c642bb 0x5c64446 0x5c64807 0x5c7e60a 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3962bc0 0x396439a 0x3804c28 0x2958f 0x2963f 0x35ec9b4 2024-09-16T16:43:19.952 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:19.964 INFO:teuthology.orchestra.run.smithi083.stderr:INFO 2024-09-16 16:43:19,965 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-16T16:43:19.964 INFO:teuthology.orchestra.run.smithi083.stderr:INFO 2024-09-16 16:43:19,965 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 2-3 2024-09-16T16:43:19.965 INFO:teuthology.orchestra.run.smithi083.stderr:DEBUG 2024-09-16 16:43:19,966 [shard 0:main] seastar - reactor::drain 2024-09-16T16:43:20.465 INFO:teuthology.orchestra.run.smithi083.stderr:----------------------------------------------------- 2024-09-16T16:43:20.465 INFO:teuthology.orchestra.run.smithi083.stderr:Suppressions used: 2024-09-16T16:43:20.465 INFO:teuthology.orchestra.run.smithi083.stderr: count bytes template 2024-09-16T16:43:20.466 INFO:teuthology.orchestra.run.smithi083.stderr: 1 8 MallocExtension::Register 2024-09-16T16:43:20.466 INFO:teuthology.orchestra.run.smithi083.stderr:----------------------------------------------------- 2024-09-16T16:43:20.466 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-16T16:43:20.517 INFO:teuthology.orchestra.run.smithi083.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-16T16:43:20.520 INFO:teuthology.orchestra.run.smithi083.stderr:INFO 2024-09-16 16:43:20,521 seastar - Reactor backend: linux-aio 2024-09-16T16:43:20.533 INFO:teuthology.orchestra.run.smithi083.stderr:==28165==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-16T16:43:20.598 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x5fb8d0b 0x5fb923a 0x5e00fef 0x5e19231 0x71579d2 0x71691a5 0x716a642 0x716a6c7 0x716a7c9 0x716a8c1 0x716aace 0x716acb4 0x716b003 0x716b3a7 0x716b5d7 0x716b841 0x716bb0e 0x716bd2c 0x716bde4 0x716beaa 0x716bf8b 0x5c63e30 0x5c63f3a 0x5c63ff6 0x5c640b2 0x5c6416e 0x5c642bb 0x5c64446 0x5c64807 0x5c7e60a 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T16:43:20.598 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:20.810 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x8b587 0x10d76c 0x10af7c 0x4225d 0x3ef7b 0xfd531 0xb85348d 0xb853540 0xba26c65 0xba26d54 0xba27414 0xba2757f 0xba27a41 0xba27e99 0xb9cba66 0xba283d8 0xba2861f 0xba286cf 0xb9d809d 0xb9dc6f8 0xb9c7696 0xb9c7a7a 0x9afa023 0x9b11b72 0x99a2168 0x9942951 0x927469d 0x932dfed 0x932ef8c 0x932f00f 0x932f094 0x932f196 0x932f28e 0x932f314 0x932f3cd 0x932f583 0x932f6bb 0x932f736 0x932f7bb 0x932f916 0x932fa30 0x932fab5 0x932fb93 0x9552886 0x961b5f3 0x961b770 0x96dc4d1 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T16:43:20.810 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:21.012 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x8b568 0x10d76c 0xdd31a 0x4ab1c 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T16:43:21.012 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:21.070 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 123 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x133d18 0xdd332 0x4ab1c 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T16:43:21.070 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:21.100 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x55422bd 0x9e3396e 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T16:43:21.100 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:21.157 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x55426f8 0x9e3397b 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T16:43:21.158 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:21.177 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x4ab7c 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T16:43:21.177 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:21.265 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 230 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x133c3f 0xdd332 0x4ab1c 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T16:43:21.265 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:21.578 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0xf9dcb 0x3869f4d 0x386a307 0xba10303 0xba27dce 0xb9cba66 0xba283d8 0xba2861f 0xba286cf 0xb9d809d 0xb9dc6f8 0xb9c7696 0xb9c7a7a 0x9afa023 0x9b11b72 0x99a2168 0x9942951 0x927469d 0x9311f16 0x93129ba 0x9312a3d 0x9312ac2 0x9312bc4 0x9312cbc 0x9312d42 0x9312dfb 0x9312fb1 0x93130e9 0x9313164 0x93131e9 0x9313344 0x931345e 0x93134e3 0x93135c1 0x9552886 0x961b5f3 0x961b770 0x96dc4d1 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T16:43:21.578 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:21.686 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x4aba4 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T16:43:21.686 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:22.696 INFO:teuthology.orchestra.run.smithi083.stderr:----------------------------------------------------- 2024-09-16T16:43:22.696 INFO:teuthology.orchestra.run.smithi083.stderr:Suppressions used: 2024-09-16T16:43:22.696 INFO:teuthology.orchestra.run.smithi083.stderr: count bytes template 2024-09-16T16:43:22.696 INFO:teuthology.orchestra.run.smithi083.stderr: 1 8 MallocExtension::Register 2024-09-16T16:43:22.696 INFO:teuthology.orchestra.run.smithi083.stderr:----------------------------------------------------- 2024-09-16T16:43:22.696 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-16T16:43:22.720 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T16:43:22.744 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T16:43:23.029 INFO:teuthology.orchestra.run.smithi083.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-16T16:43:23.030 INFO:teuthology.orchestra.run.smithi083.stderr:INFO 2024-09-16 16:43:23,030 seastar - Reactor backend: linux-aio 2024-09-16T16:43:23.035 INFO:teuthology.orchestra.run.smithi083.stderr:==28212==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-16T16:43:23.143 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x5c55b9b 0x5c55ce2 0x6038b31 0x5fb8d06 0x5fb8ee0 0x5fb946c 0x5e00fef 0x5e19231 0x71579d2 0x71691a5 0x716a642 0x716a6c7 0x716a7c9 0x716a8c1 0x716aace 0x716acb4 0x716b003 0x716b3a7 0x716b5d7 0x716b841 0x716bb0e 0x716bd2c 0x716bde4 0x716beaa 0x716bf8b 0x5c63e30 0x5c63f3a 0x5c63ff6 0x5c640b2 0x5c6416e 0x5c642bb 0x5c64446 0x5c64807 0x5c7e60a 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3962bc0 0x396439a 0x3804c28 0x2958f 0x2963f 0x35ec9b4 2024-09-16T16:43:23.143 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:23.152 INFO:teuthology.orchestra.run.smithi083.stderr:INFO 2024-09-16 16:43:23,153 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-16T16:43:23.152 INFO:teuthology.orchestra.run.smithi083.stderr:INFO 2024-09-16 16:43:23,153 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 4-5 2024-09-16T16:43:23.153 INFO:teuthology.orchestra.run.smithi083.stderr:DEBUG 2024-09-16 16:43:23,154 [shard 0:main] seastar - reactor::drain 2024-09-16T16:43:23.657 INFO:teuthology.orchestra.run.smithi083.stderr:----------------------------------------------------- 2024-09-16T16:43:23.657 INFO:teuthology.orchestra.run.smithi083.stderr:Suppressions used: 2024-09-16T16:43:23.657 INFO:teuthology.orchestra.run.smithi083.stderr: count bytes template 2024-09-16T16:43:23.657 INFO:teuthology.orchestra.run.smithi083.stderr: 1 8 MallocExtension::Register 2024-09-16T16:43:23.657 INFO:teuthology.orchestra.run.smithi083.stderr:----------------------------------------------------- 2024-09-16T16:43:23.658 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-16T16:43:23.722 INFO:teuthology.orchestra.run.smithi083.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-16T16:43:23.726 INFO:teuthology.orchestra.run.smithi083.stderr:INFO 2024-09-16 16:43:23,726 seastar - Reactor backend: linux-aio 2024-09-16T16:43:23.738 INFO:teuthology.orchestra.run.smithi083.stderr:==28210==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-16T16:43:24.013 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0xb853814 0xb8539b6 0xb853a6f 0xb853f61 0xb85436c 0xb85462b 0xba25f87 0xb9db18d 0xb9c7696 0xb9c7a7a 0x9afa023 0x9b11b72 0x99a2168 0x9942951 0x927469d 0x932dfed 0x932ef8c 0x932f00f 0x932f094 0x932f196 0x932f28e 0x932f314 0x932f3cd 0x932f583 0x932f6bb 0x932f736 0x932f7bb 0x932f916 0x932fa30 0x932fab5 0x932fb93 0x9552886 0x961b5f3 0x961b770 0x96dc4d1 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T16:43:24.013 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:24.212 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x10797d 0xdd305 0x4ab1c 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T16:43:24.212 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:24.269 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x133c3f 0xdd332 0x4ab1c 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T16:43:24.270 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:24.315 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x5542bf1 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T16:43:24.315 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:24.373 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x5542ce7 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T16:43:24.373 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:24.377 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x10798d 0xdd2fc 0x4ab1c 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T16:43:24.377 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:24.480 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 230 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x55426e3 0x9e3397b 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T16:43:24.481 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:24.774 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x10797f 0xdd2fc 0x4b14c 0x3b8c0f7 0x3b8c278 0xb9e9267 0xb9ea53a 0xb9ea7b4 0xb9ea95c 0xb9ec717 0xb9ec913 0xba317fd 0xb9d8ccf 0xb9dc6f8 0xb9c7696 0xb9c7a7a 0x9afa023 0x9b11b72 0x99a2168 0x9942951 0x927469d 0x9311f16 0x93129ba 0x9312a3d 0x9312ac2 0x9312bc4 0x9312cbc 0x9312d42 0x9312dfb 0x9312fb1 0x93130e9 0x9313164 0x93131e9 0x9313344 0x931345e 0x93134e3 0x93135c1 0x9552886 0x961b5f3 0x961b770 0x96dc4d1 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T16:43:24.775 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:24.882 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x133d18 0xdd332 0x4ab1c 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T16:43:24.883 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:24.940 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x5542c09 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T16:43:24.940 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-16T16:43:25.854 INFO:teuthology.orchestra.run.smithi083.stderr:----------------------------------------------------- 2024-09-16T16:43:25.855 INFO:teuthology.orchestra.run.smithi083.stderr:Suppressions used: 2024-09-16T16:43:25.855 INFO:teuthology.orchestra.run.smithi083.stderr: count bytes template 2024-09-16T16:43:25.855 INFO:teuthology.orchestra.run.smithi083.stderr: 1 8 MallocExtension::Register 2024-09-16T16:43:25.855 INFO:teuthology.orchestra.run.smithi083.stderr:----------------------------------------------------- 2024-09-16T16:43:25.855 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-16T16:43:25.879 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T16:43:25.903 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T16:43:25.904 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T16:43:25.904 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-16T16:43:25.968 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T16:43:25.968 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-16T16:43:26.032 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T16:43:26.033 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-16T16:43:26.098 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T16:43:26.098 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-16T16:43:26.162 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T16:43:26.163 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-16T16:43:26.219 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T16:43:26.219 DEBUG:teuthology.orchestra.run.smithi083:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T16:43:26.282 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr.x] 2024-09-16T16:43:26.282 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQAhYOhmBK02ORAAknQtE3aQsslO7rzxr4IpDw== 2024-09-16T16:43:26.282 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.0] 2024-09-16T16:43:26.282 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQAlYOhmyJoPHRAA7dlVRlDYZIkrYLWl1xugGQ== 2024-09-16T16:43:26.282 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.1] 2024-09-16T16:43:26.282 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQAoYOhmMV06KBAASN2wG3yGgCao8xUSlO+Ntg== 2024-09-16T16:43:26.282 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.2] 2024-09-16T16:43:26.282 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQArYOhmeiJQNBAAlN+t83ZfmojZUnKb4CPWZQ== 2024-09-16T16:43:26.282 INFO:teuthology.orchestra.run.smithi083.stdout:[client.0] 2024-09-16T16:43:26.282 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQAiYOhmRbzAABAA4NSzVfsxUCY/brjDPs2SeQ== 2024-09-16T16:43:26.283 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T16:43:26.385 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T16:43:26.445 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T16:43:26.506 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T16:43:26.566 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T16:43:26.630 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T16:43:26.630 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-16T16:43:26.655 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T16:43:26.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:26.764+0000 7fc1e6233c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:26.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:26.764+0000 7fc1e6233c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:26.778 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-16T16:43:26.802 DEBUG:teuthology.orchestra.run.smithi083:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T16:43:26.857 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-16T16:43:26.857 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T16:43:26.857 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T16:43:26.900 INFO:tasks.ceph.mon.a:Started 2024-09-16T16:43:26.900 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-16T16:43:26.900 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-16T16:43:26.900 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-16T16:43:26.902 INFO:tasks.ceph.mgr.x:Started 2024-09-16T16:43:26.902 DEBUG:tasks.ceph:set 0 configs 2024-09-16T16:43:26.902 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T16:43:26.903 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph osd crush tunables default 2024-09-16T16:43:26.981 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-16T16:43:26.981+0000 7fccb4d22c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:26.981 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-16T16:43:26.981+0000 7fccb4d22c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:26.982 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-16T16:43:26.981+0000 7fccb4d22c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:26.982 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-09-16T16:43:26.982+0000 7f3bf4401000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:27.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:27.005+0000 7f550f2d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:27.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:27.017+0000 7f550f2d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:27.086 INFO:teuthology.orchestra.run.smithi083.stderr:adjusted tunables profile to default 2024-09-16T16:43:27.097 INFO:tasks.ceph:check_enable_crimson: True 2024-09-16T16:43:27.097 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-09-16T16:43:27.098 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-09-16T16:43:27.189 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-09-16T16:43:27.188+0000 7f3bf4401000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:27.189 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-09-16T16:43:27.188+0000 7f3bf4401000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:27.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:27.198+0000 7fb112351640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:27.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:27.207+0000 7fb112351640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:27.287 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-16T16:43:27.287+0000 7fccab9f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-09-16T16:43:27.287 INFO:tasks.ceph.mon.a.smithi083.stderr:Please be aware that this feature is experimental, untested, 2024-09-16T16:43:27.287 INFO:tasks.ceph.mon.a.smithi083.stderr:unsupported, and may result in data corruption, data loss, 2024-09-16T16:43:27.287 INFO:tasks.ceph.mon.a.smithi083.stderr:and/or irreparable damage to your cluster. Do not use 2024-09-16T16:43:27.287 INFO:tasks.ceph.mon.a.smithi083.stderr:feature with important data. 2024-09-16T16:43:27.287 INFO:tasks.ceph.mon.a.smithi083.stderr: 2024-09-16T16:43:28.098 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-16T16:43:28.099 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T16:43:28.099 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-16T16:43:28.123 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T16:43:28.123 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-16T16:43:28.188 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T16:43:28.188 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-16T16:43:28.252 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph osd new d777a2a1-be25-4c25-bd18-561deeca94a6 0 2024-09-16T16:43:28.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:28.394+0000 7f718eed1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:28.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:28.403+0000 7f718eed1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:28.480 INFO:teuthology.orchestra.run.smithi083.stdout:0 2024-09-16T16:43:28.492 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph osd new 2a38fe11-64bb-4eac-868d-46783d009440 1 2024-09-16T16:43:28.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:28.593+0000 7fcae4308640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:28.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:28.602+0000 7fcae4308640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:28.679 INFO:teuthology.orchestra.run.smithi083.stdout:1 2024-09-16T16:43:28.691 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph osd new 899de202-15cb-4681-a007-35b6cfeed29d 2 2024-09-16T16:43:28.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:28.792+0000 7ffaf6b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:28.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:28.802+0000 7ffaf6b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:28.880 INFO:teuthology.orchestra.run.smithi083.stdout:2 2024-09-16T16:43:28.892 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-09-16T16:43:28.921 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T16:43:28.921 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-09-16T16:43:28.964 INFO:tasks.ceph.osd.0:Started 2024-09-16T16:43:28.964 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-09-16T16:43:28.992 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T16:43:28.992 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-16T16:43:29.034 INFO:tasks.ceph.osd.1:Started 2024-09-16T16:43:29.034 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-09-16T16:43:29.064 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T16:43:29.064 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-09-16T16:43:29.106 INFO:tasks.ceph.osd.2:Started 2024-09-16T16:43:29.107 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T16:43:29.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:29.272+0000 7f6552391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:29.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:29.281+0000 7f6552391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:29.350 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:29.350 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":6,"fsid":"9917636f-e891-42c5-96cb-136e608bdb04","created":"2024-09-16T16:43:27.004161+0000","modified":"2024-09-16T16:43:28.880353+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T16:43:28.880353+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"d777a2a1-be25-4c25-bd18-561deeca94a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a38fe11-64bb-4eac-868d-46783d009440","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"899de202-15cb-4681-a007-35b6cfeed29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T16:43:29.362 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-16T16:43:29.362 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T16:43:29.664 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T16:43:29.770 INFO:teuthology.misc.health.smithi083.stderr:2024-09-16T16:43:29.765+0000 7fadcb991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:29.774 INFO:teuthology.misc.health.smithi083.stderr:2024-09-16T16:43:29.774+0000 7fadcb991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:29.844 INFO:teuthology.misc.health.smithi083.stdout: 2024-09-16T16:43:29.844 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":6,"fsid":"9917636f-e891-42c5-96cb-136e608bdb04","created":"2024-09-16T16:43:27.004161+0000","modified":"2024-09-16T16:43:28.880353+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T16:43:28.880353+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"d777a2a1-be25-4c25-bd18-561deeca94a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a38fe11-64bb-4eac-868d-46783d009440","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"899de202-15cb-4681-a007-35b6cfeed29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T16:43:29.855 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-16T16:43:32.626 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-09-16T16:43:32.625+0000 7f3b9168e640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-16T16:43:36.158 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T16:43:36.260 INFO:teuthology.misc.health.smithi083.stderr:2024-09-16T16:43:36.256+0000 7f4c0ed51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:36.264 INFO:teuthology.misc.health.smithi083.stderr:2024-09-16T16:43:36.263+0000 7f4c0ed51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:36.452 INFO:teuthology.misc.health.smithi083.stdout: 2024-09-16T16:43:36.452 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"9917636f-e891-42c5-96cb-136e608bdb04","created":"2024-09-16T16:43:27.004161+0000","modified":"2024-09-16T16:43:35.690487+0000","last_up_change":"2024-09-16T16:43:34.680659+0000","last_in_change":"2024-09-16T16:43:28.880353+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T16:43:35.638595+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d777a2a1-be25-4c25-bd18-561deeca94a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1988702388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6809","nonce":1988702388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":1988702388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1988702388}]},"public_addr":"172.21.15.83:6810/1988702388","cluster_addr":"172.21.15.83:6809/1988702388","heartbeat_back_addr":"172.21.15.83:6815/1988702388","heartbeat_front_addr":"172.21.15.83:6814/1988702388","state":["exists","up"]},{"osd":1,"uuid":"2a38fe11-64bb-4eac-868d-46783d009440","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":979701936}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6805","nonce":979701936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":979701936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6803","nonce":979701936}]},"public_addr":"172.21.15.83:6804/979701936","cluster_addr":"172.21.15.83:6805/979701936","heartbeat_back_addr":"172.21.15.83:6802/979701936","heartbeat_front_addr":"172.21.15.83:6803/979701936","state":["exists","up"]},{"osd":2,"uuid":"899de202-15cb-4681-a007-35b6cfeed29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6807","nonce":1286470300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1286470300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6821","nonce":1286470300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1286470300}]},"public_addr":"172.21.15.83:6807/1286470300","cluster_addr":"172.21.15.83:6808/1286470300","heartbeat_back_addr":"172.21.15.83:6821/1286470300","heartbeat_front_addr":"172.21.15.83:6822/1286470300","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T16:43:36.465 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-16T16:43:36.465 INFO:tasks.ceph:Creating RBD pool 2024-09-16T16:43:36.465 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-16T16:43:36.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:36.564+0000 7fa3c38bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:36.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:36.575+0000 7fa3c38bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:37.704 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'rbd' created 2024-09-16T16:43:37.725 DEBUG:teuthology.orchestra.run.smithi083:> rbd --cluster ceph pool init rbd 2024-09-16T16:43:37.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:37.839+0000 7f6b0b610ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:37.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:37.840+0000 7f6b0b610ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:37.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:37.846+0000 7f6b0b610ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:40.773 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-16T16:43:40.774 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-16T16:43:40.774 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T16:43:40.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:40.877+0000 7f0a08063640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:40.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:40.886+0000 7f0a08063640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:41.071 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:41.084 INFO:teuthology.orchestra.run.smithi083.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2024-09-16T16:43:41.084 INFO:tasks.ceph_manager:config epoch is 1 2024-09-16T16:43:41.085 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T16:43:41.085 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-16T16:43:41.085 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-16T16:43:41.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:41.189+0000 7f5d9e0bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:41.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:41.197+0000 7f5d9e0bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:41.425 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:41.436 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":1643639645},{"type":"v1","addr":"172.21.15.83:6801","nonce":1643639645}]},"active_addr":"172.21.15.83:6801/1643639645","active_change":"2024-09-16T16:43:31.611782+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":2946545149}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":4234915411}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":2741182398}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":2158318025}]}]} 2024-09-16T16:43:41.440 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-16T16:43:41.440 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T16:43:41.440 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T16:43:41.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:41.541+0000 7fbfe3f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:41.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:41.548+0000 7fbfe3f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:41.736 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:41.736 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":15,"fsid":"9917636f-e891-42c5-96cb-136e608bdb04","created":"2024-09-16T16:43:27.004161+0000","modified":"2024-09-16T16:43:40.715727+0000","last_up_change":"2024-09-16T16:43:34.680659+0000","last_in_change":"2024-09-16T16:43:28.880353+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T16:43:35.638595+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T16:43:36.818038+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d777a2a1-be25-4c25-bd18-561deeca94a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1988702388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6809","nonce":1988702388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":1988702388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1988702388}]},"public_addr":"172.21.15.83:6810/1988702388","cluster_addr":"172.21.15.83:6809/1988702388","heartbeat_back_addr":"172.21.15.83:6815/1988702388","heartbeat_front_addr":"172.21.15.83:6814/1988702388","state":["exists","up"]},{"osd":1,"uuid":"2a38fe11-64bb-4eac-868d-46783d009440","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":979701936}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6805","nonce":979701936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":979701936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6803","nonce":979701936}]},"public_addr":"172.21.15.83:6804/979701936","cluster_addr":"172.21.15.83:6805/979701936","heartbeat_back_addr":"172.21.15.83:6802/979701936","heartbeat_front_addr":"172.21.15.83:6803/979701936","state":["exists","up"]},{"osd":2,"uuid":"899de202-15cb-4681-a007-35b6cfeed29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6807","nonce":1286470300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1286470300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6821","nonce":1286470300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1286470300}]},"public_addr":"172.21.15.83:6807/1286470300","cluster_addr":"172.21.15.83:6808/1286470300","heartbeat_back_addr":"172.21.15.83:6821/1286470300","heartbeat_front_addr":"172.21.15.83:6822/1286470300","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T16:43:41.749 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T16:43:41.750 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T16:43:41.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:41.851+0000 7fa739991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:41.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:41.858+0000 7fa739991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:42.045 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:42.045 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":15,"fsid":"9917636f-e891-42c5-96cb-136e608bdb04","created":"2024-09-16T16:43:27.004161+0000","modified":"2024-09-16T16:43:40.715727+0000","last_up_change":"2024-09-16T16:43:34.680659+0000","last_in_change":"2024-09-16T16:43:28.880353+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T16:43:35.638595+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T16:43:36.818038+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d777a2a1-be25-4c25-bd18-561deeca94a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1988702388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6809","nonce":1988702388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":1988702388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1988702388}]},"public_addr":"172.21.15.83:6810/1988702388","cluster_addr":"172.21.15.83:6809/1988702388","heartbeat_back_addr":"172.21.15.83:6815/1988702388","heartbeat_front_addr":"172.21.15.83:6814/1988702388","state":["exists","up"]},{"osd":1,"uuid":"2a38fe11-64bb-4eac-868d-46783d009440","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":979701936}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6805","nonce":979701936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":979701936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6803","nonce":979701936}]},"public_addr":"172.21.15.83:6804/979701936","cluster_addr":"172.21.15.83:6805/979701936","heartbeat_back_addr":"172.21.15.83:6802/979701936","heartbeat_front_addr":"172.21.15.83:6803/979701936","state":["exists","up"]},{"osd":2,"uuid":"899de202-15cb-4681-a007-35b6cfeed29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6807","nonce":1286470300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1286470300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6821","nonce":1286470300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1286470300}]},"public_addr":"172.21.15.83:6807/1286470300","cluster_addr":"172.21.15.83:6808/1286470300","heartbeat_back_addr":"172.21.15.83:6821/1286470300","heartbeat_front_addr":"172.21.15.83:6822/1286470300","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T16:43:42.058 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T16:43:42.058 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T16:43:42.059 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T16:43:42.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:42.165+0000 7fd4828fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:42.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:42.165+0000 7f1bf1693640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:42.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:42.165+0000 7f62d4cc8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:42.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:42.175+0000 7fd4828fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:42.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:42.176+0000 7f1bf1693640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:42.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:42.176+0000 7f62d4cc8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:42.209 INFO:teuthology.orchestra.run.smithi083.stdout:34359738376 2024-09-16T16:43:42.209 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T16:43:42.212 INFO:teuthology.orchestra.run.smithi083.stdout:38654705671 2024-09-16T16:43:42.212 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T16:43:42.214 INFO:teuthology.orchestra.run.smithi083.stdout:38654705671 2024-09-16T16:43:42.215 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T16:43:42.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:42.314+0000 7fcf11f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:42.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:42.323+0000 7fcf11f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:42.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:42.362+0000 7f7428d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:42.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:42.361+0000 7fdf11351640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:42.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:42.371+0000 7fdf11351640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:42.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:42.371+0000 7f7428d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:42.526 INFO:teuthology.orchestra.run.smithi083.stdout:34359738376 2024-09-16T16:43:42.538 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738376 got 34359738376 for osd.1 2024-09-16T16:43:42.539 DEBUG:teuthology.parallel:result is None 2024-09-16T16:43:42.569 INFO:teuthology.orchestra.run.smithi083.stdout:38654705671 2024-09-16T16:43:42.573 INFO:teuthology.orchestra.run.smithi083.stdout:38654705671 2024-09-16T16:43:42.583 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705671 for osd.2 2024-09-16T16:43:42.583 DEBUG:teuthology.parallel:result is None 2024-09-16T16:43:42.586 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705671 for osd.0 2024-09-16T16:43:42.586 DEBUG:teuthology.parallel:result is None 2024-09-16T16:43:42.586 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T16:43:42.586 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T16:43:42.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:42.730+0000 7f4c51591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:42.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:42.737+0000 7f4c51591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:42.920 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:42.920 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-16T16:43:42.933 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-09-16T16:43:42.203141+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3372,"kb_used_data":3372,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889812,"statfs":{"total":160658620416,"available":160655167488,"internally_reserved":0,"allocated":3452928,"data_stored":3452928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T16:43:40.741428+0000","last_change":"2024-09-16T16:43:40.748278+0000","last_active":"2024-09-16T16:43:40.741428+0000","last_peered":"2024-09-16T16:43:40.741428+0000","last_clean":"2024-09-16T16:43:40.741428+0000","last_became_active":"2024-09-16T16:43:38.775951+0000","last_became_peered":"2024-09-16T16:43:38.775951+0000","last_unstale":"2024-09-16T16:43:40.741428+0000","last_undegraded":"2024-09-16T16:43:40.741428+0000","last_fullsized":"2024-09-16T16:43:40.741428+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T16:43:40.748738+0000","last_change":"2024-09-16T16:43:40.755925+0000","last_active":"2024-09-16T16:43:40.748738+0000","last_peered":"2024-09-16T16:43:40.748738+0000","last_clean":"2024-09-16T16:43:40.748738+0000","last_became_active":"2024-09-16T16:43:38.820232+0000","last_became_peered":"2024-09-16T16:43:38.820232+0000","last_unstale":"2024-09-16T16:43:40.748738+0000","last_undegraded":"2024-09-16T16:43:40.748738+0000","last_fullsized":"2024-09-16T16:43:40.748738+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T16:43:40.740132+0000","last_change":"2024-09-16T16:43:40.748001+0000","last_active":"2024-09-16T16:43:40.740132+0000","last_peered":"2024-09-16T16:43:40.740132+0000","last_clean":"2024-09-16T16:43:40.740132+0000","last_became_active":"2024-09-16T16:43:38.794074+0000","last_became_peered":"2024-09-16T16:43:38.794074+0000","last_unstale":"2024-09-16T16:43:40.740132+0000","last_undegraded":"2024-09-16T16:43:40.740132+0000","last_fullsized":"2024-09-16T16:43:40.740132+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T16:43:40.749444+0000","last_change":"2024-09-16T16:43:40.756777+0000","last_active":"2024-09-16T16:43:40.749444+0000","last_peered":"2024-09-16T16:43:40.749444+0000","last_clean":"2024-09-16T16:43:40.749444+0000","last_became_active":"2024-09-16T16:43:38.815470+0000","last_became_peered":"2024-09-16T16:43:38.815470+0000","last_unstale":"2024-09-16T16:43:40.749444+0000","last_undegraded":"2024-09-16T16:43:40.749444+0000","last_fullsized":"2024-09-16T16:43:40.749444+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T16:43:40.752272+0000","last_change":"2024-09-16T16:43:40.752272+0000","last_active":"2024-09-16T16:43:40.752272+0000","last_peered":"2024-09-16T16:43:40.752272+0000","last_clean":"2024-09-16T16:43:40.752272+0000","last_became_active":"2024-09-16T16:43:38.806876+0000","last_became_peered":"2024-09-16T16:43:38.806876+0000","last_unstale":"2024-09-16T16:43:40.752272+0000","last_undegraded":"2024-09-16T16:43:40.752272+0000","last_fullsized":"2024-09-16T16:43:40.752272+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T16:43:40.749041+0000","last_change":"2024-09-16T16:43:40.752644+0000","last_active":"2024-09-16T16:43:40.749041+0000","last_peered":"2024-09-16T16:43:40.749041+0000","last_clean":"2024-09-16T16:43:40.749041+0000","last_became_active":"2024-09-16T16:43:38.863342+0000","last_became_peered":"2024-09-16T16:43:38.863342+0000","last_unstale":"2024-09-16T16:43:40.749041+0000","last_undegraded":"2024-09-16T16:43:40.749041+0000","last_fullsized":"2024-09-16T16:43:40.749041+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T16:43:40.744413+0000","last_change":"2024-09-16T16:43:40.747694+0000","last_active":"2024-09-16T16:43:40.744413+0000","last_peered":"2024-09-16T16:43:40.744413+0000","last_clean":"2024-09-16T16:43:40.744413+0000","last_became_active":"2024-09-16T16:43:38.835526+0000","last_became_peered":"2024-09-16T16:43:38.835526+0000","last_unstale":"2024-09-16T16:43:40.744413+0000","last_undegraded":"2024-09-16T16:43:40.744413+0000","last_fullsized":"2024-09-16T16:43:40.744413+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T16:43:40.748065+0000","last_change":"2024-09-16T16:43:40.756542+0000","last_active":"2024-09-16T16:43:40.748065+0000","last_peered":"2024-09-16T16:43:40.748065+0000","last_clean":"2024-09-16T16:43:40.748065+0000","last_became_active":"2024-09-16T16:43:38.784594+0000","last_became_peered":"2024-09-16T16:43:38.784594+0000","last_unstale":"2024-09-16T16:43:40.748065+0000","last_undegraded":"2024-09-16T16:43:40.748065+0000","last_fullsized":"2024-09-16T16:43:40.748065+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":87,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T16:43:40.743898+0000","last_change":"2024-09-16T16:43:36.785207+0000","last_active":"2024-09-16T16:43:40.743898+0000","last_peered":"2024-09-16T16:43:40.743898+0000","last_clean":"2024-09-16T16:43:40.743898+0000","last_became_active":"2024-09-16T16:43:36.756880+0000","last_became_peered":"2024-09-16T16:43:36.756880+0000","last_unstale":"2024-09-16T16:43:40.743898+0000","last_undegraded":"2024-09-16T16:43:40.743898+0000","last_fullsized":"2024-09-16T16:43:40.743898+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:35.690487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:35.690487+0000","last_clean_scrub_stamp":"2024-09-16T16:43:35.690487+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655167488,"internally_reserved":0,"allocated":3452928,"data_stored":3452928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":436,"kb_used_data":436,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297292,"statfs":{"total":53552873472,"available":53552427008,"internally_reserved":0,"allocated":446464,"data_stored":446464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738376,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1596,"kb_used_data":1596,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296132,"statfs":{"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1340,"kb_used_data":1340,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296388,"statfs":{"total":53552873472,"available":53551501312,"internally_reserved":0,"allocated":1372160,"data_stored":1372160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551501312,"internally_reserved":0,"allocated":1372160,"data_stored":1372160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552427008,"internally_reserved":0,"allocated":446464,"data_stored":446464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T16:43:42.933 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T16:43:43.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:43.035+0000 7f934ded1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:43.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:43.042+0000 7f934ded1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:43.224 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:43.224 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-16T16:43:43.236 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-09-16T16:43:42.203141+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3372,"kb_used_data":3372,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889812,"statfs":{"total":160658620416,"available":160655167488,"internally_reserved":0,"allocated":3452928,"data_stored":3452928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T16:43:40.741428+0000","last_change":"2024-09-16T16:43:40.748278+0000","last_active":"2024-09-16T16:43:40.741428+0000","last_peered":"2024-09-16T16:43:40.741428+0000","last_clean":"2024-09-16T16:43:40.741428+0000","last_became_active":"2024-09-16T16:43:38.775951+0000","last_became_peered":"2024-09-16T16:43:38.775951+0000","last_unstale":"2024-09-16T16:43:40.741428+0000","last_undegraded":"2024-09-16T16:43:40.741428+0000","last_fullsized":"2024-09-16T16:43:40.741428+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T16:43:40.748738+0000","last_change":"2024-09-16T16:43:40.755925+0000","last_active":"2024-09-16T16:43:40.748738+0000","last_peered":"2024-09-16T16:43:40.748738+0000","last_clean":"2024-09-16T16:43:40.748738+0000","last_became_active":"2024-09-16T16:43:38.820232+0000","last_became_peered":"2024-09-16T16:43:38.820232+0000","last_unstale":"2024-09-16T16:43:40.748738+0000","last_undegraded":"2024-09-16T16:43:40.748738+0000","last_fullsized":"2024-09-16T16:43:40.748738+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T16:43:40.740132+0000","last_change":"2024-09-16T16:43:40.748001+0000","last_active":"2024-09-16T16:43:40.740132+0000","last_peered":"2024-09-16T16:43:40.740132+0000","last_clean":"2024-09-16T16:43:40.740132+0000","last_became_active":"2024-09-16T16:43:38.794074+0000","last_became_peered":"2024-09-16T16:43:38.794074+0000","last_unstale":"2024-09-16T16:43:40.740132+0000","last_undegraded":"2024-09-16T16:43:40.740132+0000","last_fullsized":"2024-09-16T16:43:40.740132+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T16:43:40.749444+0000","last_change":"2024-09-16T16:43:40.756777+0000","last_active":"2024-09-16T16:43:40.749444+0000","last_peered":"2024-09-16T16:43:40.749444+0000","last_clean":"2024-09-16T16:43:40.749444+0000","last_became_active":"2024-09-16T16:43:38.815470+0000","last_became_peered":"2024-09-16T16:43:38.815470+0000","last_unstale":"2024-09-16T16:43:40.749444+0000","last_undegraded":"2024-09-16T16:43:40.749444+0000","last_fullsized":"2024-09-16T16:43:40.749444+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T16:43:40.752272+0000","last_change":"2024-09-16T16:43:40.752272+0000","last_active":"2024-09-16T16:43:40.752272+0000","last_peered":"2024-09-16T16:43:40.752272+0000","last_clean":"2024-09-16T16:43:40.752272+0000","last_became_active":"2024-09-16T16:43:38.806876+0000","last_became_peered":"2024-09-16T16:43:38.806876+0000","last_unstale":"2024-09-16T16:43:40.752272+0000","last_undegraded":"2024-09-16T16:43:40.752272+0000","last_fullsized":"2024-09-16T16:43:40.752272+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T16:43:40.749041+0000","last_change":"2024-09-16T16:43:40.752644+0000","last_active":"2024-09-16T16:43:40.749041+0000","last_peered":"2024-09-16T16:43:40.749041+0000","last_clean":"2024-09-16T16:43:40.749041+0000","last_became_active":"2024-09-16T16:43:38.863342+0000","last_became_peered":"2024-09-16T16:43:38.863342+0000","last_unstale":"2024-09-16T16:43:40.749041+0000","last_undegraded":"2024-09-16T16:43:40.749041+0000","last_fullsized":"2024-09-16T16:43:40.749041+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T16:43:40.744413+0000","last_change":"2024-09-16T16:43:40.747694+0000","last_active":"2024-09-16T16:43:40.744413+0000","last_peered":"2024-09-16T16:43:40.744413+0000","last_clean":"2024-09-16T16:43:40.744413+0000","last_became_active":"2024-09-16T16:43:38.835526+0000","last_became_peered":"2024-09-16T16:43:38.835526+0000","last_unstale":"2024-09-16T16:43:40.744413+0000","last_undegraded":"2024-09-16T16:43:40.744413+0000","last_fullsized":"2024-09-16T16:43:40.744413+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T16:43:40.748065+0000","last_change":"2024-09-16T16:43:40.756542+0000","last_active":"2024-09-16T16:43:40.748065+0000","last_peered":"2024-09-16T16:43:40.748065+0000","last_clean":"2024-09-16T16:43:40.748065+0000","last_became_active":"2024-09-16T16:43:38.784594+0000","last_became_peered":"2024-09-16T16:43:38.784594+0000","last_unstale":"2024-09-16T16:43:40.748065+0000","last_undegraded":"2024-09-16T16:43:40.748065+0000","last_fullsized":"2024-09-16T16:43:40.748065+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":87,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T16:43:40.743898+0000","last_change":"2024-09-16T16:43:36.785207+0000","last_active":"2024-09-16T16:43:40.743898+0000","last_peered":"2024-09-16T16:43:40.743898+0000","last_clean":"2024-09-16T16:43:40.743898+0000","last_became_active":"2024-09-16T16:43:36.756880+0000","last_became_peered":"2024-09-16T16:43:36.756880+0000","last_unstale":"2024-09-16T16:43:40.743898+0000","last_undegraded":"2024-09-16T16:43:40.743898+0000","last_fullsized":"2024-09-16T16:43:40.743898+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:35.690487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:35.690487+0000","last_clean_scrub_stamp":"2024-09-16T16:43:35.690487+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655167488,"internally_reserved":0,"allocated":3452928,"data_stored":3452928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":436,"kb_used_data":436,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297292,"statfs":{"total":53552873472,"available":53552427008,"internally_reserved":0,"allocated":446464,"data_stored":446464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738376,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1596,"kb_used_data":1596,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296132,"statfs":{"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1340,"kb_used_data":1340,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296388,"statfs":{"total":53552873472,"available":53551501312,"internally_reserved":0,"allocated":1372160,"data_stored":1372160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551501312,"internally_reserved":0,"allocated":1372160,"data_stored":1372160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552427008,"internally_reserved":0,"allocated":446464,"data_stored":446464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T16:43:43.237 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T16:43:43.237 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T16:43:43.237 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-16T16:43:43.237 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T16:43:43.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:43.338+0000 7ff873b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:43.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:43:43.345+0000 7ff873b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:43:43.565 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:43:43.565 INFO:teuthology.orchestra.run.smithi083.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T16:43:43.578 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-16T16:43:43.578 INFO:teuthology.run_tasks:Running task workunit... 2024-09-16T16:43:43.592 INFO:tasks.workunit:Pulling workunits from ref 0325fd3c11a225c0fd9eb38fab88aad1370ad779 2024-09-16T16:43:43.593 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-16T16:43:43.593 DEBUG:teuthology.orchestra.run.smithi083:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-16T16:43:43.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T16:43:43.608 INFO:teuthology.orchestra.run.smithi083.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T16:43:43.608 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T16:43:43.664 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-16T16:43:43.664 DEBUG:teuthology.orchestra.run.smithi083:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-16T16:43:43.720 INFO:tasks.workunit:timeout=3h 2024-09-16T16:43:43.720 INFO:tasks.workunit:cleanup=True 2024-09-16T16:43:43.721 DEBUG:teuthology.orchestra.run.smithi083:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 0325fd3c11a225c0fd9eb38fab88aad1370ad779 2024-09-16T16:43:43.777 INFO:tasks.workunit.client.0.smithi083.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-16T16:44:34.987 INFO:tasks.workunit.client.0.smithi083.stderr:Note: switching to '0325fd3c11a225c0fd9eb38fab88aad1370ad779'. 2024-09-16T16:44:34.987 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-09-16T16:44:34.987 INFO:tasks.workunit.client.0.smithi083.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-16T16:44:34.987 INFO:tasks.workunit.client.0.smithi083.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-16T16:44:34.987 INFO:tasks.workunit.client.0.smithi083.stderr:state without impacting any branches by switching back to a branch. 2024-09-16T16:44:34.987 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-09-16T16:44:34.987 INFO:tasks.workunit.client.0.smithi083.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-16T16:44:34.987 INFO:tasks.workunit.client.0.smithi083.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-16T16:44:34.987 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-09-16T16:44:34.987 INFO:tasks.workunit.client.0.smithi083.stderr: git switch -c 2024-09-16T16:44:34.987 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-09-16T16:44:34.988 INFO:tasks.workunit.client.0.smithi083.stderr:Or undo this operation with: 2024-09-16T16:44:34.988 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-09-16T16:44:34.988 INFO:tasks.workunit.client.0.smithi083.stderr: git switch - 2024-09-16T16:44:34.988 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-09-16T16:44:34.988 INFO:tasks.workunit.client.0.smithi083.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-16T16:44:34.988 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-09-16T16:44:34.988 INFO:tasks.workunit.client.0.smithi083.stderr:HEAD is now at 0325fd3c11a Merge pull request #58530 from zdover23/wip-doc-2024-07-11-radosgw-s3-authentication 2024-09-16T16:44:34.992 DEBUG:teuthology.orchestra.run.smithi083:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-16T16:44:35.060 INFO:tasks.workunit.client.0.smithi083.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-16T16:44:35.062 INFO:tasks.workunit.client.0.smithi083.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T16:44:35.062 INFO:tasks.workunit.client.0.smithi083.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-16T16:44:35.676 INFO:tasks.workunit.client.0.smithi083.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-16T16:44:35.749 INFO:tasks.workunit.client.0.smithi083.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-16T16:44:35.789 INFO:tasks.workunit.client.0.smithi083.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T16:44:35.791 INFO:tasks.workunit.client.0.smithi083.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T16:44:35.791 INFO:tasks.workunit.client.0.smithi083.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-16T16:44:35.830 INFO:tasks.workunit.client.0.smithi083.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T16:44:35.833 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T16:44:35.833 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-16T16:44:35.889 INFO:tasks.workunit:Running workunits matching rbd/test_lock_fence.sh on client.0... 2024-09-16T16:44:35.890 INFO:tasks.workunit:Running workunit rbd/test_lock_fence.sh... 2024-09-16T16:44:35.890 DEBUG:teuthology.orchestra.run.smithi083:workunit test rbd/test_lock_fence.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0325fd3c11a225c0fd9eb38fab88aad1370ad779 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_lock_fence.sh 2024-09-16T16:44:35.949 INFO:tasks.workunit.client.0.smithi083.stderr:+ IMAGE=rbdrw-image 2024-09-16T16:44:35.949 INFO:tasks.workunit.client.0.smithi083.stderr:+ LOCKID=rbdrw 2024-09-16T16:44:35.949 INFO:tasks.workunit.client.0.smithi083.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_lock_fence.sh 2024-09-16T16:44:35.950 INFO:tasks.workunit.client.0.smithi083.stderr:+ RELPATH=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/librbd 2024-09-16T16:44:35.950 INFO:tasks.workunit.client.0.smithi083.stderr:+ RBDRW=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/librbd/rbdrw.py 2024-09-16T16:44:35.950 INFO:tasks.workunit.client.0.smithi083.stderr:+ rbd create rbdrw-image --size 10 --image-format 2 --image-shared 2024-09-16T16:44:35.994 INFO:tasks.workunit.client.0.smithi083.stderr:2024-09-16T16:44:35.993+0000 7fc1a4210ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:44:35.994 INFO:tasks.workunit.client.0.smithi083.stderr:2024-09-16T16:44:35.993+0000 7fc1a4210ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:44:35.998 INFO:tasks.workunit.client.0.smithi083.stderr:2024-09-16T16:44:35.998+0000 7fc1a4210ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:44:36.096 INFO:tasks.workunit.client.0.smithi083.stderr:+ iochild=30115 2024-09-16T16:44:36.096 INFO:tasks.workunit.client.0.smithi083.stderr:+ LOCKS='[]' 2024-09-16T16:44:36.096 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' '[]' == '[]' ']' 2024-09-16T16:44:36.096 INFO:tasks.workunit.client.0.smithi083.stderr:+ python3 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/librbd/rbdrw.py rbdrw-image rbdrw 2024-09-16T16:44:36.096 INFO:tasks.workunit.client.0.smithi083.stderr:++ rbd lock list rbdrw-image --format json 2024-09-16T16:44:36.138 INFO:tasks.workunit.client.0.smithi083.stderr:2024-09-16T16:44:36.137+0000 7f1cf6802ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:44:36.138 INFO:tasks.workunit.client.0.smithi083.stderr:2024-09-16T16:44:36.137+0000 7f1cf6802ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:44:36.143 INFO:tasks.workunit.client.0.smithi083.stderr:2024-09-16T16:44:36.143+0000 7f1cf6802ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:44:36.267 INFO:tasks.workunit.client.0.smithi083.stderr:+ LOCKS='[]' 2024-09-16T16:44:36.267 INFO:tasks.workunit.client.0.smithi083.stderr:+ sleep 1 2024-09-16T16:44:37.268 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' '[]' == '[]' ']' 2024-09-16T16:44:37.268 INFO:tasks.workunit.client.0.smithi083.stderr:++ rbd lock list rbdrw-image --format json 2024-09-16T16:44:37.310 INFO:tasks.workunit.client.0.smithi083.stderr:2024-09-16T16:44:37.310+0000 7fe87e683ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:44:37.311 INFO:tasks.workunit.client.0.smithi083.stderr:2024-09-16T16:44:37.310+0000 7fe87e683ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:44:37.316 INFO:tasks.workunit.client.0.smithi083.stderr:2024-09-16T16:44:37.316+0000 7fe87e683ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:44:37.449 INFO:tasks.workunit.client.0.smithi083.stderr:+ LOCKS='[{"id":"rbdrw","locker":"client.4195","address":"172.21.15.83:0/2850401985"}]' 2024-09-16T16:44:37.449 INFO:tasks.workunit.client.0.smithi083.stderr:+ sleep 1 2024-09-16T16:44:38.450 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' '[{"id":"rbdrw","locker":"client.4195","address":"172.21.15.83:0/2850401985"}]' == '[]' ']' 2024-09-16T16:44:38.451 INFO:tasks.workunit.client.0.smithi083.stderr:++ rbd lock list rbdrw-image 2024-09-16T16:44:38.451 INFO:tasks.workunit.client.0.smithi083.stderr:++ tail -1 2024-09-16T16:44:38.451 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $NF;}' 2024-09-16T16:44:38.498 INFO:tasks.workunit.client.0.smithi083.stderr:2024-09-16T16:44:38.498+0000 7f7735874ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:44:38.498 INFO:tasks.workunit.client.0.smithi083.stderr:2024-09-16T16:44:38.498+0000 7f7735874ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:44:38.505 INFO:tasks.workunit.client.0.smithi083.stderr:2024-09-16T16:44:38.505+0000 7f7735874ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:44:38.660 INFO:tasks.workunit.client.0.smithi083.stderr:+ clientaddr=172.21.15.83:0/2850401985 2024-09-16T16:44:38.660 INFO:tasks.workunit.client.0.smithi083.stderr:++ rbd lock list rbdrw-image 2024-09-16T16:44:38.660 INFO:tasks.workunit.client.0.smithi083.stderr:++ tail -1 2024-09-16T16:44:38.660 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1;}' 2024-09-16T16:44:38.711 INFO:tasks.workunit.client.0.smithi083.stderr:2024-09-16T16:44:38.711+0000 7f4678e3bac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:44:38.711 INFO:tasks.workunit.client.0.smithi083.stderr:2024-09-16T16:44:38.711+0000 7f4678e3bac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:44:38.717 INFO:tasks.workunit.client.0.smithi083.stderr:2024-09-16T16:44:38.717+0000 7f4678e3bac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:44:38.849 INFO:tasks.workunit.client.0.smithi083.stderr:+ clientid=client.4195 2024-09-16T16:44:38.850 INFO:tasks.workunit.client.0.smithi083.stdout:clientaddr: 172.21.15.83:0/2850401985 2024-09-16T16:44:38.850 INFO:tasks.workunit.client.0.smithi083.stdout:clientid: client.4195 2024-09-16T16:44:38.850 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'clientaddr: 172.21.15.83:0/2850401985' 2024-09-16T16:44:38.850 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'clientid: client.4195' 2024-09-16T16:44:38.850 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph osd blocklist add 172.21.15.83:0/2850401985 2024-09-16T16:44:38.947 INFO:tasks.workunit.client.0.smithi083.stderr:2024-09-16T16:44:38.939+0000 7f6af8af4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:44:38.952 INFO:tasks.workunit.client.0.smithi083.stderr:2024-09-16T16:44:38.952+0000 7f6af8af4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:44:41.023 INFO:tasks.workunit.client.0.smithi083.stderr:blocklisting 172.21.15.83:0/2850401985 until 2024-09-16T17:44:40.152061+0000 (3600 sec) 2024-09-16T16:44:41.039 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 30115 2024-09-16T16:44:41.039 INFO:tasks.workunit.client.0.smithi083.stderr:+ rbdrw_exitcode=108 2024-09-16T16:44:41.039 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 108 '!=' 108 ']' 2024-09-16T16:44:41.039 INFO:tasks.workunit.client.0.smithi083.stdout:rbdrw stopped with ESHUTDOWN 2024-09-16T16:44:41.039 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'rbdrw stopped with ESHUTDOWN' 2024-09-16T16:44:41.040 INFO:tasks.workunit.client.0.smithi083.stderr:+ set -e 2024-09-16T16:44:41.040 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph osd blocklist rm 172.21.15.83:0/2850401985 2024-09-16T16:44:41.134 INFO:tasks.workunit.client.0.smithi083.stderr:2024-09-16T16:44:41.128+0000 7ff764463640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:44:41.138 INFO:tasks.workunit.client.0.smithi083.stderr:2024-09-16T16:44:41.138+0000 7ff764463640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:44:42.157 INFO:tasks.workunit.client.0.smithi083.stderr:172.21.15.83:0/2850401985 isn't blocklisted 2024-09-16T16:44:42.169 INFO:tasks.workunit.client.0.smithi083.stderr:+ rbd lock remove rbdrw-image rbdrw client.4195 2024-09-16T16:44:42.209 INFO:tasks.workunit.client.0.smithi083.stderr:2024-09-16T16:44:42.208+0000 7fb0339dfac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:44:42.209 INFO:tasks.workunit.client.0.smithi083.stderr:2024-09-16T16:44:42.209+0000 7fb0339dfac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:44:42.213 INFO:tasks.workunit.client.0.smithi083.stderr:2024-09-16T16:44:42.213+0000 7fb0339dfac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:44:43.103 INFO:tasks.workunit.client.0.smithi083.stderr:+ sleep 30 2024-09-16T16:45:13.104 INFO:tasks.workunit.client.0.smithi083.stderr:+ rbd rm rbdrw-image 2024-09-16T16:45:13.149 INFO:tasks.workunit.client.0.smithi083.stderr:2024-09-16T16:45:13.148+0000 7f290243bac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:45:13.149 INFO:tasks.workunit.client.0.smithi083.stderr:2024-09-16T16:45:13.148+0000 7f290243bac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:45:13.154 INFO:tasks.workunit.client.0.smithi083.stderr:2024-09-16T16:45:13.153+0000 7f290243bac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:45:13.759 INFO:tasks.workunit.client.0.smithi083.stderr: Removing image: 33% complete... Removing image: 66% complete... Removing image: 100% complete...done. 2024-09-16T16:45:13.763 INFO:tasks.workunit.client.0.smithi083.stdout:OK 2024-09-16T16:45:13.764 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo OK 2024-09-16T16:45:13.765 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T16:45:13.765 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T16:45:13.830 INFO:tasks.workunit:Stopping ['rbd/test_lock_fence.sh'] on client.0... 2024-09-16T16:45:13.830 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-16T16:45:14.152 DEBUG:teuthology.parallel:result is None 2024-09-16T16:45:14.153 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T16:45:14.177 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T16:45:14.177 DEBUG:teuthology.orchestra.run.smithi083:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T16:45:14.236 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T16:45:14.237 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T16:45:14.246 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T16:45:14.246 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T16:45:14.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:45:14.379+0000 7fe868288640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:45:14.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:45:14.387+0000 7fe868288640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:45:14.569 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:45:14.570 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-16T16:45:14.582 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2024-09-16T16:45:13.636482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594499,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":404,"num_read_kb":703,"num_write":246,"num_write_kb":1751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":224,"ondisk_log_size":224,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":4276,"kb_used_data":4276,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156888908,"statfs":{"total":160658620416,"available":160654241792,"internally_reserved":0,"allocated":4378624,"data_stored":4378624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4112,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":351,"num_read_kb":657,"num_write":165,"num_write_kb":643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003353"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.052154+0000","last_change":"2024-09-16T16:43:40.748278+0000","last_active":"2024-09-16T16:44:43.052154+0000","last_peered":"2024-09-16T16:44:43.052154+0000","last_clean":"2024-09-16T16:44:43.052154+0000","last_became_active":"2024-09-16T16:43:38.775951+0000","last_became_peered":"2024-09-16T16:43:38.775951+0000","last_unstale":"2024-09-16T16:44:43.052154+0000","last_undegraded":"2024-09-16T16:44:43.052154+0000","last_fullsized":"2024-09-16T16:44:43.052154+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.058293+0000","last_change":"2024-09-16T16:43:40.755925+0000","last_active":"2024-09-16T16:44:43.058293+0000","last_peered":"2024-09-16T16:44:43.058293+0000","last_clean":"2024-09-16T16:44:43.058293+0000","last_became_active":"2024-09-16T16:43:38.820232+0000","last_became_peered":"2024-09-16T16:43:38.820232+0000","last_unstale":"2024-09-16T16:44:43.058293+0000","last_undegraded":"2024-09-16T16:44:43.058293+0000","last_fullsized":"2024-09-16T16:44:43.058293+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"15'161","reported_seq":359,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.050797+0000","last_change":"2024-09-16T16:43:40.748001+0000","last_active":"2024-09-16T16:44:43.050797+0000","last_peered":"2024-09-16T16:44:43.050797+0000","last_clean":"2024-09-16T16:44:43.050797+0000","last_became_active":"2024-09-16T16:43:38.794074+0000","last_became_peered":"2024-09-16T16:43:38.794074+0000","last_unstale":"2024-09-16T16:44:43.050797+0000","last_undegraded":"2024-09-16T16:44:43.050797+0000","last_fullsized":"2024-09-16T16:44:43.050797+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4112,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":179,"num_read_kb":642,"num_write":161,"num_write_kb":641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"15'1","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.057125+0000","last_change":"2024-09-16T16:43:40.756777+0000","last_active":"2024-09-16T16:44:43.057125+0000","last_peered":"2024-09-16T16:44:43.057125+0000","last_clean":"2024-09-16T16:44:43.057125+0000","last_became_active":"2024-09-16T16:43:38.815470+0000","last_became_peered":"2024-09-16T16:43:38.815470+0000","last_unstale":"2024-09-16T16:44:43.057125+0000","last_undegraded":"2024-09-16T16:44:43.057125+0000","last_fullsized":"2024-09-16T16:44:43.057125+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'9","reported_seq":102,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.083553+0000","last_change":"2024-09-16T16:43:40.752272+0000","last_active":"2024-09-16T16:44:43.083553+0000","last_peered":"2024-09-16T16:44:43.083553+0000","last_clean":"2024-09-16T16:44:43.083553+0000","last_became_active":"2024-09-16T16:43:38.806876+0000","last_became_peered":"2024-09-16T16:43:38.806876+0000","last_unstale":"2024-09-16T16:44:43.083553+0000","last_undegraded":"2024-09-16T16:44:43.083553+0000","last_fullsized":"2024-09-16T16:44:43.083553+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":170,"num_read_kb":15,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.056640+0000","last_change":"2024-09-16T16:43:40.752644+0000","last_active":"2024-09-16T16:44:43.056640+0000","last_peered":"2024-09-16T16:44:43.056640+0000","last_clean":"2024-09-16T16:44:43.056640+0000","last_became_active":"2024-09-16T16:43:38.863342+0000","last_became_peered":"2024-09-16T16:43:38.863342+0000","last_unstale":"2024-09-16T16:44:43.056640+0000","last_undegraded":"2024-09-16T16:44:43.056640+0000","last_fullsized":"2024-09-16T16:44:43.056640+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.054458+0000","last_change":"2024-09-16T16:43:40.747694+0000","last_active":"2024-09-16T16:44:43.054458+0000","last_peered":"2024-09-16T16:44:43.054458+0000","last_clean":"2024-09-16T16:44:43.054458+0000","last_became_active":"2024-09-16T16:43:38.835526+0000","last_became_peered":"2024-09-16T16:43:38.835526+0000","last_unstale":"2024-09-16T16:44:43.054458+0000","last_undegraded":"2024-09-16T16:44:43.054458+0000","last_fullsized":"2024-09-16T16:44:43.054458+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.055710+0000","last_change":"2024-09-16T16:43:40.756542+0000","last_active":"2024-09-16T16:44:43.055710+0000","last_peered":"2024-09-16T16:44:43.055710+0000","last_clean":"2024-09-16T16:44:43.055710+0000","last_became_active":"2024-09-16T16:43:38.784594+0000","last_became_peered":"2024-09-16T16:43:38.784594+0000","last_unstale":"2024-09-16T16:44:43.055710+0000","last_undegraded":"2024-09-16T16:44:43.055710+0000","last_fullsized":"2024-09-16T16:44:43.055710+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":99,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.053510+0000","last_change":"2024-09-16T16:43:36.785207+0000","last_active":"2024-09-16T16:44:43.053510+0000","last_peered":"2024-09-16T16:44:43.053510+0000","last_clean":"2024-09-16T16:44:43.053510+0000","last_became_active":"2024-09-16T16:43:36.756880+0000","last_became_peered":"2024-09-16T16:43:36.756880+0000","last_unstale":"2024-09-16T16:44:43.053510+0000","last_undegraded":"2024-09-16T16:44:43.053510+0000","last_fullsized":"2024-09-16T16:44:43.053510+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:35.690487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:35.690487+0000","last_clean_scrub_stamp":"2024-09-16T16:43:35.690487+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4131,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":351,"num_read_kb":657,"num_write":167,"num_write_kb":645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160654241792,"internally_reserved":0,"allocated":4378624,"data_stored":4378624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":172,"ondisk_log_size":172,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53550780416,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":468,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297260,"statfs":{"total":53552873472,"available":53552394240,"internally_reserved":0,"allocated":479232,"data_stored":479232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2044,"kb_used_data":2044,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295684,"statfs":{"total":53552873472,"available":53550780416,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1764,"kb_used_data":1764,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295964,"statfs":{"total":53552873472,"available":53551067136,"internally_reserved":0,"allocated":1806336,"data_stored":1806336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53550780416,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551067136,"internally_reserved":0,"allocated":1806336,"data_stored":1806336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550780416,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552394240,"internally_reserved":0,"allocated":479232,"data_stored":479232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T16:45:14.583 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T16:45:14.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:45:14.686+0000 7fcb49f51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:45:14.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:45:14.693+0000 7fcb49f51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:45:14.901 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:45:14.901 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-16T16:45:14.914 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2024-09-16T16:45:13.636482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594499,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":404,"num_read_kb":703,"num_write":246,"num_write_kb":1751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":224,"ondisk_log_size":224,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":4276,"kb_used_data":4276,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156888908,"statfs":{"total":160658620416,"available":160654241792,"internally_reserved":0,"allocated":4378624,"data_stored":4378624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4112,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":351,"num_read_kb":657,"num_write":165,"num_write_kb":643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003353"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.052154+0000","last_change":"2024-09-16T16:43:40.748278+0000","last_active":"2024-09-16T16:44:43.052154+0000","last_peered":"2024-09-16T16:44:43.052154+0000","last_clean":"2024-09-16T16:44:43.052154+0000","last_became_active":"2024-09-16T16:43:38.775951+0000","last_became_peered":"2024-09-16T16:43:38.775951+0000","last_unstale":"2024-09-16T16:44:43.052154+0000","last_undegraded":"2024-09-16T16:44:43.052154+0000","last_fullsized":"2024-09-16T16:44:43.052154+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.058293+0000","last_change":"2024-09-16T16:43:40.755925+0000","last_active":"2024-09-16T16:44:43.058293+0000","last_peered":"2024-09-16T16:44:43.058293+0000","last_clean":"2024-09-16T16:44:43.058293+0000","last_became_active":"2024-09-16T16:43:38.820232+0000","last_became_peered":"2024-09-16T16:43:38.820232+0000","last_unstale":"2024-09-16T16:44:43.058293+0000","last_undegraded":"2024-09-16T16:44:43.058293+0000","last_fullsized":"2024-09-16T16:44:43.058293+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"15'161","reported_seq":359,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.050797+0000","last_change":"2024-09-16T16:43:40.748001+0000","last_active":"2024-09-16T16:44:43.050797+0000","last_peered":"2024-09-16T16:44:43.050797+0000","last_clean":"2024-09-16T16:44:43.050797+0000","last_became_active":"2024-09-16T16:43:38.794074+0000","last_became_peered":"2024-09-16T16:43:38.794074+0000","last_unstale":"2024-09-16T16:44:43.050797+0000","last_undegraded":"2024-09-16T16:44:43.050797+0000","last_fullsized":"2024-09-16T16:44:43.050797+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4112,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":179,"num_read_kb":642,"num_write":161,"num_write_kb":641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"15'1","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.057125+0000","last_change":"2024-09-16T16:43:40.756777+0000","last_active":"2024-09-16T16:44:43.057125+0000","last_peered":"2024-09-16T16:44:43.057125+0000","last_clean":"2024-09-16T16:44:43.057125+0000","last_became_active":"2024-09-16T16:43:38.815470+0000","last_became_peered":"2024-09-16T16:43:38.815470+0000","last_unstale":"2024-09-16T16:44:43.057125+0000","last_undegraded":"2024-09-16T16:44:43.057125+0000","last_fullsized":"2024-09-16T16:44:43.057125+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'9","reported_seq":102,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.083553+0000","last_change":"2024-09-16T16:43:40.752272+0000","last_active":"2024-09-16T16:44:43.083553+0000","last_peered":"2024-09-16T16:44:43.083553+0000","last_clean":"2024-09-16T16:44:43.083553+0000","last_became_active":"2024-09-16T16:43:38.806876+0000","last_became_peered":"2024-09-16T16:43:38.806876+0000","last_unstale":"2024-09-16T16:44:43.083553+0000","last_undegraded":"2024-09-16T16:44:43.083553+0000","last_fullsized":"2024-09-16T16:44:43.083553+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":170,"num_read_kb":15,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.056640+0000","last_change":"2024-09-16T16:43:40.752644+0000","last_active":"2024-09-16T16:44:43.056640+0000","last_peered":"2024-09-16T16:44:43.056640+0000","last_clean":"2024-09-16T16:44:43.056640+0000","last_became_active":"2024-09-16T16:43:38.863342+0000","last_became_peered":"2024-09-16T16:43:38.863342+0000","last_unstale":"2024-09-16T16:44:43.056640+0000","last_undegraded":"2024-09-16T16:44:43.056640+0000","last_fullsized":"2024-09-16T16:44:43.056640+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.054458+0000","last_change":"2024-09-16T16:43:40.747694+0000","last_active":"2024-09-16T16:44:43.054458+0000","last_peered":"2024-09-16T16:44:43.054458+0000","last_clean":"2024-09-16T16:44:43.054458+0000","last_became_active":"2024-09-16T16:43:38.835526+0000","last_became_peered":"2024-09-16T16:43:38.835526+0000","last_unstale":"2024-09-16T16:44:43.054458+0000","last_undegraded":"2024-09-16T16:44:43.054458+0000","last_fullsized":"2024-09-16T16:44:43.054458+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.055710+0000","last_change":"2024-09-16T16:43:40.756542+0000","last_active":"2024-09-16T16:44:43.055710+0000","last_peered":"2024-09-16T16:44:43.055710+0000","last_clean":"2024-09-16T16:44:43.055710+0000","last_became_active":"2024-09-16T16:43:38.784594+0000","last_became_peered":"2024-09-16T16:43:38.784594+0000","last_unstale":"2024-09-16T16:44:43.055710+0000","last_undegraded":"2024-09-16T16:44:43.055710+0000","last_fullsized":"2024-09-16T16:44:43.055710+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":99,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.053510+0000","last_change":"2024-09-16T16:43:36.785207+0000","last_active":"2024-09-16T16:44:43.053510+0000","last_peered":"2024-09-16T16:44:43.053510+0000","last_clean":"2024-09-16T16:44:43.053510+0000","last_became_active":"2024-09-16T16:43:36.756880+0000","last_became_peered":"2024-09-16T16:43:36.756880+0000","last_unstale":"2024-09-16T16:44:43.053510+0000","last_undegraded":"2024-09-16T16:44:43.053510+0000","last_fullsized":"2024-09-16T16:44:43.053510+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:35.690487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:35.690487+0000","last_clean_scrub_stamp":"2024-09-16T16:43:35.690487+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4131,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":351,"num_read_kb":657,"num_write":167,"num_write_kb":645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160654241792,"internally_reserved":0,"allocated":4378624,"data_stored":4378624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":172,"ondisk_log_size":172,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53550780416,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":468,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297260,"statfs":{"total":53552873472,"available":53552394240,"internally_reserved":0,"allocated":479232,"data_stored":479232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2044,"kb_used_data":2044,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295684,"statfs":{"total":53552873472,"available":53550780416,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1764,"kb_used_data":1764,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295964,"statfs":{"total":53552873472,"available":53551067136,"internally_reserved":0,"allocated":1806336,"data_stored":1806336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53550780416,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551067136,"internally_reserved":0,"allocated":1806336,"data_stored":1806336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550780416,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552394240,"internally_reserved":0,"allocated":479232,"data_stored":479232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T16:45:14.915 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T16:45:14.915 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T16:45:15.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:45:15.018+0000 7f2e62a63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:45:15.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:45:15.025+0000 7f2e62a63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:45:15.212 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:45:15.213 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-16T16:45:15.225 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2024-09-16T16:45:13.636482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594499,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":404,"num_read_kb":703,"num_write":246,"num_write_kb":1751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":224,"ondisk_log_size":224,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":4276,"kb_used_data":4276,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156888908,"statfs":{"total":160658620416,"available":160654241792,"internally_reserved":0,"allocated":4378624,"data_stored":4378624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4112,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":351,"num_read_kb":657,"num_write":165,"num_write_kb":643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003353"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.052154+0000","last_change":"2024-09-16T16:43:40.748278+0000","last_active":"2024-09-16T16:44:43.052154+0000","last_peered":"2024-09-16T16:44:43.052154+0000","last_clean":"2024-09-16T16:44:43.052154+0000","last_became_active":"2024-09-16T16:43:38.775951+0000","last_became_peered":"2024-09-16T16:43:38.775951+0000","last_unstale":"2024-09-16T16:44:43.052154+0000","last_undegraded":"2024-09-16T16:44:43.052154+0000","last_fullsized":"2024-09-16T16:44:43.052154+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.058293+0000","last_change":"2024-09-16T16:43:40.755925+0000","last_active":"2024-09-16T16:44:43.058293+0000","last_peered":"2024-09-16T16:44:43.058293+0000","last_clean":"2024-09-16T16:44:43.058293+0000","last_became_active":"2024-09-16T16:43:38.820232+0000","last_became_peered":"2024-09-16T16:43:38.820232+0000","last_unstale":"2024-09-16T16:44:43.058293+0000","last_undegraded":"2024-09-16T16:44:43.058293+0000","last_fullsized":"2024-09-16T16:44:43.058293+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"15'161","reported_seq":359,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.050797+0000","last_change":"2024-09-16T16:43:40.748001+0000","last_active":"2024-09-16T16:44:43.050797+0000","last_peered":"2024-09-16T16:44:43.050797+0000","last_clean":"2024-09-16T16:44:43.050797+0000","last_became_active":"2024-09-16T16:43:38.794074+0000","last_became_peered":"2024-09-16T16:43:38.794074+0000","last_unstale":"2024-09-16T16:44:43.050797+0000","last_undegraded":"2024-09-16T16:44:43.050797+0000","last_fullsized":"2024-09-16T16:44:43.050797+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4112,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":179,"num_read_kb":642,"num_write":161,"num_write_kb":641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"15'1","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.057125+0000","last_change":"2024-09-16T16:43:40.756777+0000","last_active":"2024-09-16T16:44:43.057125+0000","last_peered":"2024-09-16T16:44:43.057125+0000","last_clean":"2024-09-16T16:44:43.057125+0000","last_became_active":"2024-09-16T16:43:38.815470+0000","last_became_peered":"2024-09-16T16:43:38.815470+0000","last_unstale":"2024-09-16T16:44:43.057125+0000","last_undegraded":"2024-09-16T16:44:43.057125+0000","last_fullsized":"2024-09-16T16:44:43.057125+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'9","reported_seq":102,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.083553+0000","last_change":"2024-09-16T16:43:40.752272+0000","last_active":"2024-09-16T16:44:43.083553+0000","last_peered":"2024-09-16T16:44:43.083553+0000","last_clean":"2024-09-16T16:44:43.083553+0000","last_became_active":"2024-09-16T16:43:38.806876+0000","last_became_peered":"2024-09-16T16:43:38.806876+0000","last_unstale":"2024-09-16T16:44:43.083553+0000","last_undegraded":"2024-09-16T16:44:43.083553+0000","last_fullsized":"2024-09-16T16:44:43.083553+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":170,"num_read_kb":15,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.056640+0000","last_change":"2024-09-16T16:43:40.752644+0000","last_active":"2024-09-16T16:44:43.056640+0000","last_peered":"2024-09-16T16:44:43.056640+0000","last_clean":"2024-09-16T16:44:43.056640+0000","last_became_active":"2024-09-16T16:43:38.863342+0000","last_became_peered":"2024-09-16T16:43:38.863342+0000","last_unstale":"2024-09-16T16:44:43.056640+0000","last_undegraded":"2024-09-16T16:44:43.056640+0000","last_fullsized":"2024-09-16T16:44:43.056640+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.054458+0000","last_change":"2024-09-16T16:43:40.747694+0000","last_active":"2024-09-16T16:44:43.054458+0000","last_peered":"2024-09-16T16:44:43.054458+0000","last_clean":"2024-09-16T16:44:43.054458+0000","last_became_active":"2024-09-16T16:43:38.835526+0000","last_became_peered":"2024-09-16T16:43:38.835526+0000","last_unstale":"2024-09-16T16:44:43.054458+0000","last_undegraded":"2024-09-16T16:44:43.054458+0000","last_fullsized":"2024-09-16T16:44:43.054458+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.055710+0000","last_change":"2024-09-16T16:43:40.756542+0000","last_active":"2024-09-16T16:44:43.055710+0000","last_peered":"2024-09-16T16:44:43.055710+0000","last_clean":"2024-09-16T16:44:43.055710+0000","last_became_active":"2024-09-16T16:43:38.784594+0000","last_became_peered":"2024-09-16T16:43:38.784594+0000","last_unstale":"2024-09-16T16:44:43.055710+0000","last_undegraded":"2024-09-16T16:44:43.055710+0000","last_fullsized":"2024-09-16T16:44:43.055710+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":99,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.053510+0000","last_change":"2024-09-16T16:43:36.785207+0000","last_active":"2024-09-16T16:44:43.053510+0000","last_peered":"2024-09-16T16:44:43.053510+0000","last_clean":"2024-09-16T16:44:43.053510+0000","last_became_active":"2024-09-16T16:43:36.756880+0000","last_became_peered":"2024-09-16T16:43:36.756880+0000","last_unstale":"2024-09-16T16:44:43.053510+0000","last_undegraded":"2024-09-16T16:44:43.053510+0000","last_fullsized":"2024-09-16T16:44:43.053510+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:35.690487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:35.690487+0000","last_clean_scrub_stamp":"2024-09-16T16:43:35.690487+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4131,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":351,"num_read_kb":657,"num_write":167,"num_write_kb":645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160654241792,"internally_reserved":0,"allocated":4378624,"data_stored":4378624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":172,"ondisk_log_size":172,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53550780416,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":468,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297260,"statfs":{"total":53552873472,"available":53552394240,"internally_reserved":0,"allocated":479232,"data_stored":479232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2044,"kb_used_data":2044,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295684,"statfs":{"total":53552873472,"available":53550780416,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1764,"kb_used_data":1764,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295964,"statfs":{"total":53552873472,"available":53551067136,"internally_reserved":0,"allocated":1806336,"data_stored":1806336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53550780416,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551067136,"internally_reserved":0,"allocated":1806336,"data_stored":1806336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550780416,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552394240,"internally_reserved":0,"allocated":479232,"data_stored":479232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T16:45:15.226 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T16:45:15.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:45:15.329+0000 7f76afd91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:45:15.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:45:15.335+0000 7f76afd91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:45:15.521 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:45:15.522 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":19,"fsid":"9917636f-e891-42c5-96cb-136e608bdb04","created":"2024-09-16T16:43:27.004161+0000","modified":"2024-09-16T16:44:43.031265+0000","last_up_change":"2024-09-16T16:43:34.680659+0000","last_in_change":"2024-09-16T16:43:28.880353+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T16:43:35.638595+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T16:43:36.818038+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d777a2a1-be25-4c25-bd18-561deeca94a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1988702388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6809","nonce":1988702388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":1988702388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1988702388}]},"public_addr":"172.21.15.83:6810/1988702388","cluster_addr":"172.21.15.83:6809/1988702388","heartbeat_back_addr":"172.21.15.83:6815/1988702388","heartbeat_front_addr":"172.21.15.83:6814/1988702388","state":["exists","up"]},{"osd":1,"uuid":"2a38fe11-64bb-4eac-868d-46783d009440","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":979701936}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6805","nonce":979701936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":979701936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6803","nonce":979701936}]},"public_addr":"172.21.15.83:6804/979701936","cluster_addr":"172.21.15.83:6805/979701936","heartbeat_back_addr":"172.21.15.83:6802/979701936","heartbeat_front_addr":"172.21.15.83:6803/979701936","state":["exists","up"]},{"osd":2,"uuid":"899de202-15cb-4681-a007-35b6cfeed29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6807","nonce":1286470300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1286470300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6821","nonce":1286470300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1286470300}]},"public_addr":"172.21.15.83:6807/1286470300","cluster_addr":"172.21.15.83:6808/1286470300","heartbeat_back_addr":"172.21.15.83:6821/1286470300","heartbeat_front_addr":"172.21.15.83:6822/1286470300","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:0/2850401985":"2024-09-16T17:44:42.338275+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T16:45:16.536 INFO:tasks.ceph:Scrubbing osd.0 2024-09-16T16:45:16.536 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T16:45:16.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:45:16.649+0000 7f326e69c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:45:16.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:45:16.661+0000 7f326e69c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:45:16.676 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-16T16:45:16.676 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "" 2024-09-16T16:45:16.676 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-16T16:45:16.687 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-16T16:45:16.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:45:16.792+0000 7f816a751640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:45:16.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:45:16.800+0000 7f816a751640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:45:16.988 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T16:45:17.007 INFO:tasks.ceph:Scrubbing osd.1 2024-09-16T16:45:17.007 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T16:45:17.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:45:17.111+0000 7f56beb51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:45:17.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:45:17.123+0000 7f56beb51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:45:17.136 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-16T16:45:17.136 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "" 2024-09-16T16:45:17.136 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-16T16:45:17.148 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-16T16:45:17.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:45:17.251+0000 7f65f8191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:45:17.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:45:17.259+0000 7f65f8191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:45:17.442 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T16:45:17.458 INFO:tasks.ceph:Scrubbing osd.2 2024-09-16T16:45:17.458 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T16:45:17.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:45:17.568+0000 7f8d57cd1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:45:17.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:45:17.580+0000 7f8d57cd1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:45:17.594 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-16T16:45:17.594 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "" 2024-09-16T16:45:17.594 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-16T16:45:17.605 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-16T16:45:17.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:45:17.708+0000 7f7a1d991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:45:17.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:45:17.717+0000 7f7a1d991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:45:17.904 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T16:45:17.922 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T16:45:18.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:45:18.024+0000 7f3f64d51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:45:18.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:45:18.031+0000 7f3f64d51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:45:18.216 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:45:18.216 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-16T16:45:18.229 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":64,"stamp":"2024-09-16T16:45:17.637329+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594499,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":404,"num_read_kb":703,"num_write":246,"num_write_kb":1751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":224,"ondisk_log_size":224,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":4276,"kb_used_data":4276,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156888908,"statfs":{"total":160658620416,"available":160654241792,"internally_reserved":0,"allocated":4378624,"data_stored":4378624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003132"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.052154+0000","last_change":"2024-09-16T16:43:40.748278+0000","last_active":"2024-09-16T16:44:43.052154+0000","last_peered":"2024-09-16T16:44:43.052154+0000","last_clean":"2024-09-16T16:44:43.052154+0000","last_became_active":"2024-09-16T16:43:38.775951+0000","last_became_peered":"2024-09-16T16:43:38.775951+0000","last_unstale":"2024-09-16T16:44:43.052154+0000","last_undegraded":"2024-09-16T16:44:43.052154+0000","last_fullsized":"2024-09-16T16:44:43.052154+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.058293+0000","last_change":"2024-09-16T16:43:40.755925+0000","last_active":"2024-09-16T16:44:43.058293+0000","last_peered":"2024-09-16T16:44:43.058293+0000","last_clean":"2024-09-16T16:44:43.058293+0000","last_became_active":"2024-09-16T16:43:38.820232+0000","last_became_peered":"2024-09-16T16:43:38.820232+0000","last_unstale":"2024-09-16T16:44:43.058293+0000","last_undegraded":"2024-09-16T16:44:43.058293+0000","last_fullsized":"2024-09-16T16:44:43.058293+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"15'161","reported_seq":359,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.050797+0000","last_change":"2024-09-16T16:43:40.748001+0000","last_active":"2024-09-16T16:44:43.050797+0000","last_peered":"2024-09-16T16:44:43.050797+0000","last_clean":"2024-09-16T16:44:43.050797+0000","last_became_active":"2024-09-16T16:43:38.794074+0000","last_became_peered":"2024-09-16T16:43:38.794074+0000","last_unstale":"2024-09-16T16:44:43.050797+0000","last_undegraded":"2024-09-16T16:44:43.050797+0000","last_fullsized":"2024-09-16T16:44:43.050797+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4112,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":179,"num_read_kb":642,"num_write":161,"num_write_kb":641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"15'1","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.057125+0000","last_change":"2024-09-16T16:43:40.756777+0000","last_active":"2024-09-16T16:44:43.057125+0000","last_peered":"2024-09-16T16:44:43.057125+0000","last_clean":"2024-09-16T16:44:43.057125+0000","last_became_active":"2024-09-16T16:43:38.815470+0000","last_became_peered":"2024-09-16T16:43:38.815470+0000","last_unstale":"2024-09-16T16:44:43.057125+0000","last_undegraded":"2024-09-16T16:44:43.057125+0000","last_fullsized":"2024-09-16T16:44:43.057125+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'9","reported_seq":102,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.083553+0000","last_change":"2024-09-16T16:43:40.752272+0000","last_active":"2024-09-16T16:44:43.083553+0000","last_peered":"2024-09-16T16:44:43.083553+0000","last_clean":"2024-09-16T16:44:43.083553+0000","last_became_active":"2024-09-16T16:43:38.806876+0000","last_became_peered":"2024-09-16T16:43:38.806876+0000","last_unstale":"2024-09-16T16:44:43.083553+0000","last_undegraded":"2024-09-16T16:44:43.083553+0000","last_fullsized":"2024-09-16T16:44:43.083553+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":170,"num_read_kb":15,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.056640+0000","last_change":"2024-09-16T16:43:40.752644+0000","last_active":"2024-09-16T16:44:43.056640+0000","last_peered":"2024-09-16T16:44:43.056640+0000","last_clean":"2024-09-16T16:44:43.056640+0000","last_became_active":"2024-09-16T16:43:38.863342+0000","last_became_peered":"2024-09-16T16:43:38.863342+0000","last_unstale":"2024-09-16T16:44:43.056640+0000","last_undegraded":"2024-09-16T16:44:43.056640+0000","last_fullsized":"2024-09-16T16:44:43.056640+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.054458+0000","last_change":"2024-09-16T16:43:40.747694+0000","last_active":"2024-09-16T16:44:43.054458+0000","last_peered":"2024-09-16T16:44:43.054458+0000","last_clean":"2024-09-16T16:44:43.054458+0000","last_became_active":"2024-09-16T16:43:38.835526+0000","last_became_peered":"2024-09-16T16:43:38.835526+0000","last_unstale":"2024-09-16T16:44:43.054458+0000","last_undegraded":"2024-09-16T16:44:43.054458+0000","last_fullsized":"2024-09-16T16:44:43.054458+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.055710+0000","last_change":"2024-09-16T16:43:40.756542+0000","last_active":"2024-09-16T16:44:43.055710+0000","last_peered":"2024-09-16T16:44:43.055710+0000","last_clean":"2024-09-16T16:44:43.055710+0000","last_became_active":"2024-09-16T16:43:38.784594+0000","last_became_peered":"2024-09-16T16:43:38.784594+0000","last_unstale":"2024-09-16T16:44:43.055710+0000","last_undegraded":"2024-09-16T16:44:43.055710+0000","last_fullsized":"2024-09-16T16:44:43.055710+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:37.700289+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":99,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:44:43.053510+0000","last_change":"2024-09-16T16:43:36.785207+0000","last_active":"2024-09-16T16:44:43.053510+0000","last_peered":"2024-09-16T16:44:43.053510+0000","last_clean":"2024-09-16T16:44:43.053510+0000","last_became_active":"2024-09-16T16:43:36.756880+0000","last_became_peered":"2024-09-16T16:43:36.756880+0000","last_unstale":"2024-09-16T16:44:43.053510+0000","last_undegraded":"2024-09-16T16:44:43.053510+0000","last_fullsized":"2024-09-16T16:44:43.053510+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:43:35.690487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:43:35.690487+0000","last_clean_scrub_stamp":"2024-09-16T16:43:35.690487+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4131,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":351,"num_read_kb":657,"num_write":167,"num_write_kb":645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160654241792,"internally_reserved":0,"allocated":4378624,"data_stored":4378624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":172,"ondisk_log_size":172,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53550780416,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":468,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297260,"statfs":{"total":53552873472,"available":53552394240,"internally_reserved":0,"allocated":479232,"data_stored":479232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2044,"kb_used_data":2044,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295684,"statfs":{"total":53552873472,"available":53550780416,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1764,"kb_used_data":1764,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295964,"statfs":{"total":53552873472,"available":53551067136,"internally_reserved":0,"allocated":1806336,"data_stored":1806336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53550780416,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551067136,"internally_reserved":0,"allocated":1806336,"data_stored":1806336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550780416,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552394240,"internally_reserved":0,"allocated":479232,"data_stored":479232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T16:45:18.233 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T16:43:37.700289+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=43, tm_sec=37, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=45, tm_sec=15, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T16:45:18.233 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T16:43:37.700289+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=43, tm_sec=37, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=45, tm_sec=15, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T16:45:18.233 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T16:43:37.700289+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=43, tm_sec=37, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=45, tm_sec=15, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T16:45:18.234 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T16:43:37.700289+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=43, tm_sec=37, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=45, tm_sec=15, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T16:45:18.234 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T16:43:37.700289+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=43, tm_sec=37, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=45, tm_sec=15, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T16:45:18.234 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T16:43:37.700289+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=43, tm_sec=37, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=45, tm_sec=15, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T16:45:18.234 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T16:43:37.700289+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=43, tm_sec=37, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=45, tm_sec=15, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T16:45:18.234 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T16:43:37.700289+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=43, tm_sec=37, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=45, tm_sec=15, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T16:45:18.234 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T16:43:35.690487+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=43, tm_sec=35, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=45, tm_sec=15, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T16:45:18.234 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T16:45:38.235 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T16:45:38.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:45:38.352+0000 7f8911151640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:45:38.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:45:38.359+0000 7f8911151640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:45:38.541 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T16:45:38.541 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-16T16:45:38.554 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-09-16T16:45:37.642722+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":493,"num_read_kb":709,"num_write":252,"num_write_kb":1754,"num_scrub_errors":0,"num_shallow_scrub_errors":2,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":243,"ondisk_log_size":243,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3876,"kb_used_data":3876,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889308,"statfs":{"total":160658620416,"available":160654651392,"internally_reserved":0,"allocated":3969024,"data_stored":3969024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4112,"num_objects":-3,"num_object_clones":0,"num_object_copies":-6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":89,"num_read_kb":6,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":2,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003202"},"pg_stats":[{"pgid":"2.7","version":"19'1","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:45:17.448978+0000","last_change":"2024-09-16T16:45:17.481878+0000","last_active":"2024-09-16T16:45:17.448978+0000","last_peered":"2024-09-16T16:45:17.448978+0000","last_clean":"2024-09-16T16:45:17.448978+0000","last_became_active":"2024-09-16T16:43:38.775951+0000","last_became_peered":"2024-09-16T16:43:38.775951+0000","last_unstale":"2024-09-16T16:45:17.448978+0000","last_undegraded":"2024-09-16T16:45:17.448978+0000","last_fullsized":"2024-09-16T16:45:17.448978+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'1","last_scrub_stamp":"2024-09-16T16:45:17.481687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:45:17.481687+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"19'1","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:45:17.449006+0000","last_change":"2024-09-16T16:45:17.486442+0000","last_active":"2024-09-16T16:45:17.449006+0000","last_peered":"2024-09-16T16:45:17.449006+0000","last_clean":"2024-09-16T16:45:17.449006+0000","last_became_active":"2024-09-16T16:43:38.820232+0000","last_became_peered":"2024-09-16T16:43:38.820232+0000","last_unstale":"2024-09-16T16:45:17.449006+0000","last_undegraded":"2024-09-16T16:45:17.449006+0000","last_fullsized":"2024-09-16T16:45:17.449006+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'1","last_scrub_stamp":"2024-09-16T16:45:17.486199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:45:17.486199+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"19'164","reported_seq":363,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:45:17.447430+0000","last_change":"2024-09-16T16:45:17.479147+0000","last_active":"2024-09-16T16:45:17.447430+0000","last_peered":"2024-09-16T16:45:17.447430+0000","last_clean":"2024-09-16T16:45:17.447430+0000","last_became_active":"2024-09-16T16:43:38.794074+0000","last_became_peered":"2024-09-16T16:43:38.794074+0000","last_unstale":"2024-09-16T16:45:17.447430+0000","last_undegraded":"2024-09-16T16:45:17.447430+0000","last_fullsized":"2024-09-16T16:45:17.447430+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'164","last_scrub_stamp":"2024-09-16T16:45:17.478976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:45:17.478976+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":179,"num_read_kb":642,"num_write":163,"num_write_kb":641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"19'2","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:45:17.447460+0000","last_change":"2024-09-16T16:45:17.491102+0000","last_active":"2024-09-16T16:45:17.447460+0000","last_peered":"2024-09-16T16:45:17.447460+0000","last_clean":"2024-09-16T16:45:17.447460+0000","last_became_active":"2024-09-16T16:43:38.815470+0000","last_became_peered":"2024-09-16T16:43:38.815470+0000","last_unstale":"2024-09-16T16:45:17.447460+0000","last_undegraded":"2024-09-16T16:45:17.447460+0000","last_fullsized":"2024-09-16T16:45:17.447460+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2024-09-16T16:45:17.490915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:45:17.490915+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":1,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'17","reported_seq":143,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:45:16.990915+0000","last_change":"2024-09-16T16:45:17.009698+0000","last_active":"2024-09-16T16:45:16.990915+0000","last_peered":"2024-09-16T16:45:16.990915+0000","last_clean":"2024-09-16T16:45:16.990915+0000","last_became_active":"2024-09-16T16:43:38.806876+0000","last_became_peered":"2024-09-16T16:43:38.806876+0000","last_unstale":"2024-09-16T16:45:16.990915+0000","last_undegraded":"2024-09-16T16:45:16.990915+0000","last_fullsized":"2024-09-16T16:45:16.990915+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'17","last_scrub_stamp":"2024-09-16T16:45:17.009482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:45:17.009482+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":248,"num_read_kb":21,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:45:17.909459+0000","last_change":"2024-09-16T16:45:17.922586+0000","last_active":"2024-09-16T16:45:17.909459+0000","last_peered":"2024-09-16T16:45:17.909459+0000","last_clean":"2024-09-16T16:45:17.909459+0000","last_became_active":"2024-09-16T16:43:38.863342+0000","last_became_peered":"2024-09-16T16:43:38.863342+0000","last_unstale":"2024-09-16T16:45:17.909459+0000","last_undegraded":"2024-09-16T16:45:17.909459+0000","last_fullsized":"2024-09-16T16:45:17.909459+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:45:17.922376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:45:17.922376+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:45:17.906412+0000","last_change":"2024-09-16T16:45:17.919376+0000","last_active":"2024-09-16T16:45:17.906412+0000","last_peered":"2024-09-16T16:45:17.906412+0000","last_clean":"2024-09-16T16:45:17.906412+0000","last_became_active":"2024-09-16T16:43:38.835526+0000","last_became_peered":"2024-09-16T16:43:38.835526+0000","last_unstale":"2024-09-16T16:45:17.906412+0000","last_undegraded":"2024-09-16T16:45:17.906412+0000","last_fullsized":"2024-09-16T16:45:17.906412+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:45:17.919199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:45:17.919199+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"19'6","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:45:17.446121+0000","last_change":"2024-09-16T16:45:17.473471+0000","last_active":"2024-09-16T16:45:17.446121+0000","last_peered":"2024-09-16T16:45:17.446121+0000","last_clean":"2024-09-16T16:45:17.446121+0000","last_became_active":"2024-09-16T16:43:38.784594+0000","last_became_peered":"2024-09-16T16:43:38.784594+0000","last_unstale":"2024-09-16T16:45:17.446121+0000","last_undegraded":"2024-09-16T16:45:17.446121+0000","last_fullsized":"2024-09-16T16:45:17.446121+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'6","last_scrub_stamp":"2024-09-16T16:45:17.473279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:45:17.473279+0000","last_clean_scrub_stamp":"2024-09-16T16:43:37.700289+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":1,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":101,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T16:45:17.444903+0000","last_change":"2024-09-16T16:45:17.497648+0000","last_active":"2024-09-16T16:45:17.444903+0000","last_peered":"2024-09-16T16:45:17.444903+0000","last_clean":"2024-09-16T16:45:17.444903+0000","last_became_active":"2024-09-16T16:43:36.756880+0000","last_became_peered":"2024-09-16T16:43:36.756880+0000","last_unstale":"2024-09-16T16:45:17.444903+0000","last_undegraded":"2024-09-16T16:45:17.444903+0000","last_fullsized":"2024-09-16T16:45:17.444903+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"12'52","last_scrub_stamp":"2024-09-16T16:45:17.497463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:45:17.497463+0000","last_clean_scrub_stamp":"2024-09-16T16:43:35.690487+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":440,"num_read_kb":663,"num_write":173,"num_write_kb":648,"num_scrub_errors":0,"num_shallow_scrub_errors":2,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160654651392,"internally_reserved":0,"allocated":3969024,"data_stored":3969024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":191,"ondisk_log_size":191,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53550985216,"internally_reserved":0,"allocated":1888256,"data_stored":1888256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705783,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":532,"kb_used_data":532,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297196,"statfs":{"total":53552873472,"available":53552328704,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738487,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1844,"kb_used_data":1844,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295884,"statfs":{"total":53552873472,"available":53550985216,"internally_reserved":0,"allocated":1888256,"data_stored":1888256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705783,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1500,"kb_used_data":1500,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296228,"statfs":{"total":53552873472,"available":53551337472,"internally_reserved":0,"allocated":1536000,"data_stored":1536000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53550985216,"internally_reserved":0,"allocated":1888256,"data_stored":1888256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551337472,"internally_reserved":0,"allocated":1536000,"data_stored":1536000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550985216,"internally_reserved":0,"allocated":1888256,"data_stored":1888256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552328704,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T16:45:38.555 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-16T16:45:38.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:45:38.655+0000 7ff660791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:45:38.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T16:45:38.666+0000 7ff660791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:45:38.884 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T16:45:38.884 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T16:45:38.885 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T16:45:38.885 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T16:45:38.942 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T16:45:38.942 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T16:45:38.942 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T16:45:39.008 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T16:45:39.008 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T16:45:39.009 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T16:45:39.065 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T16:45:39.065 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T16:45:39.066 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-16T16:45:39.066 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T16:45:39.094 INFO:tasks.ceph.mgr.x:Stopped 2024-09-16T16:45:39.094 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T16:45:39.094 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T16:45:39.094 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T16:45:39.146 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T16:45:39.146 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T16:45:39.146 DEBUG:teuthology.orchestra.run.smithi083:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-16T16:45:39.212 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi083.front.sepia.ceph.com 2024-09-16T16:45:39.212 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-16T16:45:39.406 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi083.front.sepia.ceph.com 2024-09-16T16:45:39.406 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-16T16:45:39.498 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi083.front.sepia.ceph.com 2024-09-16T16:45:39.498 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-16T16:45:39.589 INFO:tasks.ceph:Archiving mon data... 2024-09-16T16:45:39.590 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2024-09-16_16:23:48-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi/7907376/data/mon.a.tgz 2024-09-16T16:45:39.590 DEBUG:teuthology.orchestra.run.smithi083:> mktemp 2024-09-16T16:45:39.605 INFO:teuthology.orchestra.run.smithi083.stdout:/tmp/tmp.nvQRb9RhMT 2024-09-16T16:45:39.606 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.nvQRb9RhMT 2024-09-16T16:45:39.708 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0666 /tmp/tmp.nvQRb9RhMT 2024-09-16T16:45:39.790 DEBUG:teuthology.orchestra.remote:smithi083:/tmp/tmp.nvQRb9RhMT is 166KB 2024-09-16T16:45:39.842 DEBUG:teuthology.orchestra.run.smithi083:> rm -fr /tmp/tmp.nvQRb9RhMT 2024-09-16T16:45:39.857 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T16:45:39.857 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T16:45:39.976 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T16:45:39.977 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T16:45:39.978 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-09-16_16:23:48-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi/7907376/remote/smithi083/crash 2024-09-16T16:45:39.979 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T16:45:40.041 INFO:tasks.ceph:Compressing logs... 2024-09-16T16:45:40.041 DEBUG:teuthology.orchestra.run.smithi083:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T16:45:40.105 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-16T16:45:40.105 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30402.log 2024-09-16T16:45:40.124 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28596.log 2024-09-16T16:45:40.124 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.30402.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30402.log.gz 2024-09-16T16:45:40.124 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30292.log 2024-09-16T16:45:40.124 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.28596.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-16T16:45:40.124 INFO:teuthology.orchestra.run.smithi083.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28596.log.gz 2024-09-16T16:45:40.124 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.30292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30654.log 2024-09-16T16:45:40.125 INFO:teuthology.orchestra.run.smithi083.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.30292.log.gz 2024-09-16T16:45:40.125 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29417.log 2024-09-16T16:45:40.125 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.30654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29073.log 2024-09-16T16:45:40.125 INFO:teuthology.orchestra.run.smithi083.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30654.log.gz 2024-09-16T16:45:40.125 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.29417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30223.log 2024-09-16T16:45:40.125 INFO:teuthology.orchestra.run.smithi083.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29417.log.gz 2024-09-16T16:45:40.125 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.29073.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29073.log.gz 2024-09-16T16:45:40.125 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28794.log 2024-09-16T16:45:40.125 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.30223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30098.log 2024-09-16T16:45:40.125 INFO:teuthology.orchestra.run.smithi083.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30223.log.gz 2024-09-16T16:45:40.125 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.28794.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28794.log.gz 2024-09-16T16:45:40.125 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30904.log 2024-09-16T16:45:40.126 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.30098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29582.log 2024-09-16T16:45:40.126 INFO:teuthology.orchestra.run.smithi083.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.30098.log.gz 2024-09-16T16:45:40.126 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.30904.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30904.log.gz 2024-09-16T16:45:40.126 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30822.log 2024-09-16T16:45:40.126 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.29582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30116.log 2024-09-16T16:45:40.126 INFO:teuthology.orchestra.run.smithi083.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29582.log.gz 2024-09-16T16:45:40.126 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.30822.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30822.log.gz 2024-09-16T16:45:40.126 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30178.log 2024-09-16T16:45:40.126 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.30116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30612.log 2024-09-16T16:45:40.126 INFO:teuthology.orchestra.run.smithi083.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.30116.log.gz 2024-09-16T16:45:40.126 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.30178.log: gzip 66.1% -- replaced with /var/log/ceph/ceph-client.admin.30178.log.gz -5 2024-09-16T16:45:40.126 INFO:teuthology.orchestra.run.smithi083.stderr: --verbose -- /var/log/ceph/ceph-client.admin.30528.log 2024-09-16T16:45:40.127 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.30612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28885.log 2024-09-16T16:45:40.127 INFO:teuthology.orchestra.run.smithi083.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30612.log.gz 2024-09-16T16:45:40.127 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.30528.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30528.log.gz 2024-09-16T16:45:40.127 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T16:45:40.127 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.28885.log: 49.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.28885.log.gz -5 2024-09-16T16:45:40.127 INFO:teuthology.orchestra.run.smithi083.stderr: --verbose -- /var/log/ceph/ceph-client.admin.30570.log 2024-09-16T16:45:40.127 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29323.log 2024-09-16T16:45:40.127 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.30570.log: 86.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T16:45:40.127 INFO:teuthology.orchestra.run.smithi083.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30570.log.gz 2024-09-16T16:45:40.127 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29268.log 2024-09-16T16:45:40.127 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29709.log 2024-09-16T16:45:40.127 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.29323.log: 65.6% -- replaced with /var/log/ceph/ceph-client.admin.29323.log.gz 2024-09-16T16:45:40.127 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.29268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29375.log 2024-09-16T16:45:40.127 INFO:teuthology.orchestra.run.smithi083.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29268.log.gz 2024-09-16T16:45:40.127 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.29709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29795.log 2024-09-16T16:45:40.128 INFO:teuthology.orchestra.run.smithi083.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29709.log.gz 2024-09-16T16:45:40.128 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.29375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29837.log 2024-09-16T16:45:40.128 INFO:teuthology.orchestra.run.smithi083.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29375.log.gz 2024-09-16T16:45:40.128 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.29795.log: gzip -5 --verbose -- 50.0% /var/log/ceph/ceph-mon.a.log 2024-09-16T16:45:40.128 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29795.log.gz 2024-09-16T16:45:40.128 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.29837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29581.log 2024-09-16T16:45:40.128 INFO:teuthology.orchestra.run.smithi083.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29837.log.gz 2024-09-16T16:45:40.128 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29043.log 2024-09-16T16:45:40.128 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.29581.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29581.log.gz 2024-09-16T16:45:40.128 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30486.log 2024-09-16T16:45:40.128 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-16T16:45:40.128 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.29043.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29043.log.gz 2024-09-16T16:45:40.129 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.30486.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30486.log.gz 2024-09-16T16:45:40.129 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28834.log 2024-09-16T16:45:40.129 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30780.log 2024-09-16T16:45:40.129 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.28834.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.28834.log.gz 2024-09-16T16:45:40.129 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T16:45:40.129 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30156.log 2024-09-16T16:45:40.129 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.30780.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30780.log.gz 2024-09-16T16:45:40.129 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29501.log 2024-09-16T16:45:40.129 INFO:teuthology.orchestra.run.smithi083.stderr: 86.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T16:45:40.129 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5/var/log/ceph/ceph-client.admin.30156.log: --verbose -- /var/log/ceph/ceph-client.admin.30271.log 2024-09-16T16:45:40.129 INFO:teuthology.orchestra.run.smithi083.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.30156.log.gz 2024-09-16T16:45:40.129 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.29501.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-16T16:45:40.129 INFO:teuthology.orchestra.run.smithi083.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29501.log.gz 2024-09-16T16:45:40.129 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.30271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29669.log 2024-09-16T16:45:40.130 INFO:teuthology.orchestra.run.smithi083.stderr: 65.6% -- replaced with /var/log/ceph/ceph-client.admin.30271.log.gz 2024-09-16T16:45:40.130 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30115.log 2024-09-16T16:45:40.130 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.29669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30444.log 2024-09-16T16:45:40.130 INFO:teuthology.orchestra.run.smithi083.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29669.log.gz 2024-09-16T16:45:40.130 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.30115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30864.log 2024-09-16T16:45:40.130 INFO:teuthology.orchestra.run.smithi083.stderr: 65.3% -- replaced with /var/log/ceph/ceph-client.admin.30115.log.gz 2024-09-16T16:45:40.130 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30201.log 2024-09-16T16:45:40.130 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.30444.log: /var/log/ceph/ceph-client.admin.30864.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30444.log.gz 2024-09-16T16:45:40.130 INFO:teuthology.orchestra.run.smithi083.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30864.log.gz 2024-09-16T16:45:40.130 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30696.log 2024-09-16T16:45:40.130 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.30201.log: gzip -5 66.7% --verbose -- /var/log/ceph/ceph-client.admin.29708.log 2024-09-16T16:45:40.130 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30201.log.gz 2024-09-16T16:45:40.130 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.30696.log: gzip -5 --verbose 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30696.log.gz 2024-09-16T16:45:40.130 INFO:teuthology.orchestra.run.smithi083.stderr: -- /var/log/ceph/ceph-client.admin.29212.log 2024-09-16T16:45:40.131 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.29708.log: 49.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28684.log 2024-09-16T16:45:40.131 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29708.log.gz 2024-09-16T16:45:40.131 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.29212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30247.log 2024-09-16T16:45:40.131 INFO:teuthology.orchestra.run.smithi083.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29212.log.gz 2024-09-16T16:45:40.131 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5/var/log/ceph/ceph-client.admin.28684.log: --verbose -- /var/log/ceph/ceph-client.admin.29583.log 2024-09-16T16:45:40.131 INFO:teuthology.orchestra.run.smithi083.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28684.log.gz 2024-09-16T16:45:40.131 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.30247.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30247.log.gz 2024-09-16T16:45:40.131 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29879.log 2024-09-16T16:45:40.131 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.29583.log: gzip -5 --verbose 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29583.log.gz 2024-09-16T16:45:40.131 INFO:teuthology.orchestra.run.smithi083.stderr: -- /var/log/ceph/ceph.tmp-client.admin.27636.log 2024-09-16T16:45:40.131 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.29879.log: 49.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30738.log 2024-09-16T16:45:40.131 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29879.log.gz 2024-09-16T16:45:40.131 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph.tmp-client.admin.27636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29459.log 2024-09-16T16:45:40.131 INFO:teuthology.orchestra.run.smithi083.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27636.log.gz 2024-09-16T16:45:40.132 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.30738.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30738.log.gz 2024-09-16T16:45:40.132 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.29459.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29459.log.gz 2024-09-16T16:45:40.132 INFO:teuthology.orchestra.run.smithi083.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-16T16:45:40.132 INFO:teuthology.orchestra.run.smithi083.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-16T16:45:40.146 INFO:teuthology.orchestra.run.smithi083.stderr: 95.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-16T16:45:40.162 INFO:teuthology.orchestra.run.smithi083.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-16T16:45:40.187 INFO:teuthology.orchestra.run.smithi083.stderr: 95.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-16T16:45:40.189 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-16T16:45:40.189 INFO:teuthology.orchestra.run.smithi083.stderr:real 0m0.093s 2024-09-16T16:45:40.189 INFO:teuthology.orchestra.run.smithi083.stderr:user 0m0.210s 2024-09-16T16:45:40.189 INFO:teuthology.orchestra.run.smithi083.stderr:sys 0m0.047s 2024-09-16T16:45:40.189 INFO:tasks.ceph:Archiving logs... 2024-09-16T16:45:40.189 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/log/ceph to /home/teuthworker/archive/matan-2024-09-16_16:23:48-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi/7907376/remote/smithi083/log 2024-09-16T16:45:40.190 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T16:45:40.333 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T16:45:40.346 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T16:45:40.346 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T16:45:40.397 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T16:45:40.406 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T16:45:40.406 DEBUG:teuthology.orchestra.run.smithi083:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T16:45:40.425 INFO:teuthology.orchestra.run.smithi083.stderr:bash: line 1: ntpq: command not found 2024-09-16T16:45:40.430 INFO:teuthology.orchestra.run.smithi083.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T16:45:40.430 INFO:teuthology.orchestra.run.smithi083.stdout:=============================================================================== 2024-09-16T16:45:40.430 INFO:teuthology.orchestra.run.smithi083.stdout:^* hv01.front.sepia.ceph.com 3 6 377 5 -418us[ -409us] +/- 39ms 2024-09-16T16:45:40.430 INFO:teuthology.orchestra.run.smithi083.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 6 +99us[ +108us] +/- 56ms 2024-09-16T16:45:40.430 INFO:teuthology.orchestra.run.smithi083.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 +785us[ +793us] +/- 76ms 2024-09-16T16:45:40.430 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T16:45:40.430 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T16:45:40.438 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T16:45:40.439 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T16:45:40.446 DEBUG:teuthology.orchestra.run.smithi083:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T16:45:40.532 DEBUG:teuthology.orchestra.run.smithi083:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T16:45:40.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T16:45:40.559 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T16:45:40.569 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T16:45:40.576 INFO:teuthology.task.internal:Duration was 657.356223 seconds 2024-09-16T16:45:40.577 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T16:45:40.585 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T16:45:40.585 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T16:45:40.637 INFO:teuthology.orchestra.run.smithi083.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T16:45:40.987 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T16:45:40.987 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi083.front.sepia.ceph.com 2024-09-16T16:45:40.987 DEBUG:teuthology.orchestra.run.smithi083:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T16:45:41.047 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T16:45:41.047 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T16:45:41.113 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T16:45:41.113 DEBUG:teuthology.orchestra.run.smithi083:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T16:45:41.252 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T16:45:41.261 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T16:45:41.261 DEBUG:teuthology.orchestra.run.smithi083:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T16:45:41.286 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T16:45:41.295 DEBUG:teuthology.orchestra.run.smithi083:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:45:41.349 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = core 2024-09-16T16:45:41.362 DEBUG:teuthology.orchestra.run.smithi083:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:45:41.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T16:45:41.417 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T16:45:41.426 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T16:45:41.426 DEBUG:teuthology.misc:Transferring archived files from smithi083:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2024-09-16_16:23:48-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi/7907376/remote/smithi083 2024-09-16T16:45:41.427 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T16:45:41.491 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T16:45:41.491 DEBUG:teuthology.orchestra.run.smithi083:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T16:45:41.540 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T16:45:41.551 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T16:45:41.551 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T16:45:41.561 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T16:45:41.561 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T16:45:41.595 INFO:teuthology.orchestra.run.smithi083.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 16:45 /home/ubuntu/cephtest 2024-09-16T16:45:41.596 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T16:45:41.605 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T16:45:41.625 INFO:teuthology.run:Summary data: description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_lock_and_fence} duration: 657.3562233448029 flavor: crimson owner: scheduled_matan@teuthology success: true 2024-09-16T16:45:41.626 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T16:45:41.710 INFO:teuthology.run:pass