2024-09-21T21:21:59.321 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-21T21:21:59.336 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T21:21:59.421 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914351 branch: main description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_lock_and_fence} email: ceph-qa@ceph.com first_in_suite: false job_id: '7914351' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd pool default crimson: true osd: crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug ms: 1 debug osd: 20 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore ompap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-1 osd.1: crimson seastar cpu cores: 2-3 osd.2: crimson seastar cpu cores: 4-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: crimson sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 2950 sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 sleep_before_teardown: 0 subset: 0/1 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 targets: smithi129.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJdvidMRrCtv9KN6zDI2TAzZamUmqkbYF/hKtDscnnAIwTQ34U488rNsfrdxy5tTdzgS7O5dzMaijm3a5/SSBI4= tasks: - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 flavor: crimson - workunit: clients: client.0: - rbd/test_lock_fence.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-21_20:56:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-21T21:21:59.421 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa; will attempt to use it 2024-09-21T21:21:59.421 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa/tasks 2024-09-21T21:21:59.422 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-21T21:21:59.423 INFO:teuthology.task.internal:Checking packages... 2024-09-21T21:21:59.443 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '62da54b4b83a96ca9976b7e1d0c8272a564a2208' 2024-09-21T21:21:59.443 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T21:21:59.443 INFO:teuthology.packaging:ref: None 2024-09-21T21:21:59.443 INFO:teuthology.packaging:tag: None 2024-09-21T21:21:59.443 INFO:teuthology.packaging:branch: main 2024-09-21T21:21:59.443 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T21:21:59.444 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=main 2024-09-21T21:21:59.642 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-5092.g62da54b4 2024-09-21T21:21:59.644 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-21T21:21:59.652 INFO:teuthology.task.internal:no buildpackages task found 2024-09-21T21:21:59.652 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-21T21:21:59.689 INFO:teuthology.task.internal:Saving configuration 2024-09-21T21:21:59.701 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-21T21:21:59.715 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-21T21:21:59.741 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914351', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 21:16:54.124345', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJdvidMRrCtv9KN6zDI2TAzZamUmqkbYF/hKtDscnnAIwTQ34U488rNsfrdxy5tTdzgS7O5dzMaijm3a5/SSBI4='} 2024-09-21T21:21:59.741 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-21T21:21:59.747 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-21T21:21:59.747 INFO:teuthology.run_tasks:Running task console_log... 2024-09-21T21:21:59.799 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fcfa88f4160>, signals=[15]) 2024-09-21T21:21:59.799 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-21T21:21:59.805 INFO:teuthology.task.internal:Opening connections... 2024-09-21T21:21:59.805 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi129.front.sepia.ceph.com 2024-09-21T21:21:59.807 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:21:59.884 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-21T21:21:59.890 DEBUG:teuthology.orchestra.run.smithi129:> uname -m 2024-09-21T21:21:59.965 INFO:teuthology.orchestra.run.smithi129.stdout:x86_64 2024-09-21T21:21:59.965 DEBUG:teuthology.orchestra.run.smithi129:> cat /etc/os-release 2024-09-21T21:22:00.023 INFO:teuthology.orchestra.run.smithi129.stdout:NAME="CentOS Stream" 2024-09-21T21:22:00.023 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION="9" 2024-09-21T21:22:00.023 INFO:teuthology.orchestra.run.smithi129.stdout:ID="centos" 2024-09-21T21:22:00.023 INFO:teuthology.orchestra.run.smithi129.stdout:ID_LIKE="rhel fedora" 2024-09-21T21:22:00.023 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_ID="9" 2024-09-21T21:22:00.023 INFO:teuthology.orchestra.run.smithi129.stdout:PLATFORM_ID="platform:el9" 2024-09-21T21:22:00.024 INFO:teuthology.orchestra.run.smithi129.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T21:22:00.024 INFO:teuthology.orchestra.run.smithi129.stdout:ANSI_COLOR="0;31" 2024-09-21T21:22:00.024 INFO:teuthology.orchestra.run.smithi129.stdout:LOGO="fedora-logo-icon" 2024-09-21T21:22:00.024 INFO:teuthology.orchestra.run.smithi129.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T21:22:00.024 INFO:teuthology.orchestra.run.smithi129.stdout:HOME_URL="https://centos.org/" 2024-09-21T21:22:00.024 INFO:teuthology.orchestra.run.smithi129.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T21:22:00.024 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T21:22:00.024 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T21:22:00.025 INFO:teuthology.lock.ops:Updating smithi129.front.sepia.ceph.com on lock server 2024-09-21T21:22:00.040 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-21T21:22:00.049 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-21T21:22:00.074 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-21T21:22:00.074 DEBUG:teuthology.orchestra.run.smithi129:> test '!' -e /home/ubuntu/cephtest 2024-09-21T21:22:00.092 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-21T21:22:00.097 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-21T21:22:00.097 DEBUG:teuthology.orchestra.run.smithi129:> test -z $(ls -A /var/lib/ceph) 2024-09-21T21:22:00.253 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-21T21:22:00.286 INFO:teuthology.run_tasks:Running task kernel... 2024-09-21T21:22:00.299 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-21T21:22:00.299 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-21T21:22:00.299 DEBUG:teuthology.orchestra.run.smithi129:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T21:22:00.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T21:22:00.317 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-09-21T21:22:00.375 INFO:teuthology.orchestra.run.smithi129.stdout:5.14.0-503.el9.x86_64 2024-09-21T21:22:00.375 INFO:teuthology.task.kernel:Running kernel on smithi129: 5.14.0-503.el9.x86_64 2024-09-21T21:22:00.375 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum install -y kernel 2024-09-21T21:22:02.903 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.3 MB 00:00 2024-09-21T21:22:04.846 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - AppStream 23 MB/s | 20 MB 00:00 2024-09-21T21:22:10.213 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 6.5 MB 00:01 2024-09-21T21:22:11.664 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - Extras packages 85 kB/s | 19 kB 00:00 2024-09-21T21:22:12.768 INFO:teuthology.orchestra.run.smithi129.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-21T21:22:18.240 INFO:teuthology.orchestra.run.smithi129.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-09-21T21:22:20.113 INFO:teuthology.orchestra.run.smithi129.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T21:22:20.114 INFO:teuthology.orchestra.run.smithi129.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T21:22:20.214 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T21:22:20.223 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:22:20.223 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2024-09-21T21:22:20.223 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:22:20.224 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T21:22:20.224 INFO:teuthology.orchestra.run.smithi129.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T21:22:20.224 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-21T21:22:20.224 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T21:22:20.224 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T21:22:20.224 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T21:22:20.224 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:22:20.224 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T21:22:20.224 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:22:20.224 INFO:teuthology.orchestra.run.smithi129.stdout:Install 4 Packages 2024-09-21T21:22:20.224 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:22:20.225 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 79 M 2024-09-21T21:22:20.225 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 126 M 2024-09-21T21:22:20.225 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T21:22:20.650 INFO:teuthology.orchestra.run.smithi129.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 122 kB/s | 36 kB 00:00 2024-09-21T21:22:22.025 INFO:teuthology.orchestra.run.smithi129.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 9.4 MB/s | 16 MB 00:01 2024-09-21T21:22:22.292 INFO:teuthology.orchestra.run.smithi129.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 17 MB/s | 29 MB 00:01 2024-09-21T21:22:23.401 INFO:teuthology.orchestra.run.smithi129.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 11 MB/s | 35 MB 00:03 2024-09-21T21:22:23.401 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:22:23.401 INFO:teuthology.orchestra.run.smithi129.stdout:Total 25 MB/s | 79 MB 00:03 2024-09-21T21:22:24.081 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T21:22:24.434 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T21:22:24.435 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T21:22:25.402 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T21:22:25.402 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T21:22:27.867 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T21:22:28.511 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T21:22:28.592 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T21:22:31.358 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T21:22:31.485 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T21:22:36.304 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T21:22:36.346 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T21:22:41.209 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T21:23:19.346 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T21:23:19.386 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T21:23:19.728 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T21:23:19.728 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T21:23:19.728 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T21:23:19.728 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T21:23:20.583 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T21:23:20.583 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:23:20.583 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T21:23:20.583 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T21:23:20.583 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T21:23:20.583 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T21:23:20.583 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T21:23:20.583 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:23:20.583 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T21:23:20.916 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T21:23:21.001 INFO:teuthology.orchestra.run.smithi129.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T21:23:21.001 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T21:23:21.001 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T21:23:21.001 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T21:23:21.002 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T21:23:21.002 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-21T21:23:21.002 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi129.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T21:23:21.002 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T21:23:21.060 INFO:teuthology.orchestra.run.smithi129.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T21:23:21.060 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T21:23:21.061 DEBUG:teuthology.orchestra.run.smithi129:> sudo rpm -qi grub2-tools 2024-09-21T21:23:21.102 INFO:teuthology.orchestra.run.smithi129.stdout:Name : grub2-tools 2024-09-21T21:23:21.102 INFO:teuthology.orchestra.run.smithi129.stdout:Epoch : 1 2024-09-21T21:23:21.102 INFO:teuthology.orchestra.run.smithi129.stdout:Version : 2.06 2024-09-21T21:23:21.102 INFO:teuthology.orchestra.run.smithi129.stdout:Release : 82.el9 2024-09-21T21:23:21.102 INFO:teuthology.orchestra.run.smithi129.stdout:Architecture: x86_64 2024-09-21T21:23:21.102 INFO:teuthology.orchestra.run.smithi129.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T21:23:21.102 INFO:teuthology.orchestra.run.smithi129.stdout:Group : Unspecified 2024-09-21T21:23:21.102 INFO:teuthology.orchestra.run.smithi129.stdout:Size : 8274670 2024-09-21T21:23:21.102 INFO:teuthology.orchestra.run.smithi129.stdout:License : GPLv3+ 2024-09-21T21:23:21.102 INFO:teuthology.orchestra.run.smithi129.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T21:23:21.103 INFO:teuthology.orchestra.run.smithi129.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T21:23:21.103 INFO:teuthology.orchestra.run.smithi129.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T21:23:21.103 INFO:teuthology.orchestra.run.smithi129.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T21:23:21.103 INFO:teuthology.orchestra.run.smithi129.stdout:Packager : builder@centos.org 2024-09-21T21:23:21.103 INFO:teuthology.orchestra.run.smithi129.stdout:Vendor : CentOS 2024-09-21T21:23:21.103 INFO:teuthology.orchestra.run.smithi129.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T21:23:21.103 INFO:teuthology.orchestra.run.smithi129.stdout:Summary : Support tools for GRUB. 2024-09-21T21:23:21.103 INFO:teuthology.orchestra.run.smithi129.stdout:Description : 2024-09-21T21:23:21.103 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:23:21.103 INFO:teuthology.orchestra.run.smithi129.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T21:23:21.103 INFO:teuthology.orchestra.run.smithi129.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T21:23:21.103 INFO:teuthology.orchestra.run.smithi129.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T21:23:21.103 INFO:teuthology.orchestra.run.smithi129.stdout:hardware devices. 2024-09-21T21:23:21.103 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:23:21.104 INFO:teuthology.orchestra.run.smithi129.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T21:23:21.104 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T21:23:21.104 INFO:teuthology.task.kernel:Updating grub on smithi129 to boot 5.14.0-508.el9.x86_64 2024-09-21T21:23:21.104 DEBUG:teuthology.orchestra.run.smithi129:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T21:23:21.761 INFO:teuthology.orchestra.run.smithi129.stderr:Generating grub configuration file ... 2024-09-21T21:23:23.077 INFO:teuthology.orchestra.run.smithi129.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T21:23:23.111 INFO:teuthology.orchestra.run.smithi129.stderr:done 2024-09-21T21:23:23.113 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2024-09-21T21:23:23.129 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.zZuioYKvwe 2024-09-21T21:23:23.129 DEBUG:teuthology.orchestra.run.smithi129:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.zZuioYKvwe 2024-09-21T21:23:23.203 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.zZuioYKvwe 2024-09-21T21:23:23.342 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.zZuioYKvwe is 6KB 2024-09-21T21:23:23.391 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.zZuioYKvwe 2024-09-21T21:23:23.406 DEBUG:teuthology.orchestra.run.smithi129:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T21:23:23.468 INFO:teuthology.orchestra.run.smithi129.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T21:23:23.468 INFO:teuthology.orchestra.run.smithi129.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T21:23:23.468 INFO:teuthology.orchestra.run.smithi129.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T21:23:23.469 INFO:teuthology.orchestra.run.smithi129.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T21:23:23.470 DEBUG:teuthology.orchestra.run.smithi129:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T21:23:23.621 DEBUG:teuthology.orchestra.run.smithi129:> sudo shutdown -r now 2024-09-21T21:23:53.650 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-21T21:23:53.650 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-21T21:23:53.651 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:24:12.143 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-09-21T21:24:21.152 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-21T21:24:21.152 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:24:24.207 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-09-21T21:24:36.218 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-21T21:24:36.219 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:24:54.639 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-09-21T21:25:09.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-21T21:25:09.640 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:25:10.044 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-09-21T21:25:10.446 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-21T21:25:10.446 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-21T21:25:10.446 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-09-21T21:25:10.461 INFO:teuthology.orchestra.run.smithi129.stdout:5.14.0-508.el9.x86_64 2024-09-21T21:25:10.461 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T21:25:10.461 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T21:25:10.461 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T21:25:11.462 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-21T21:25:11.462 DEBUG:teuthology.orchestra.run.smithi129:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T21:25:11.541 INFO:teuthology.orchestra.run.smithi129.stdout:ttyS1 2024-09-21T21:25:11.567 DEBUG:teuthology.parallel:result is None 2024-09-21T21:25:11.567 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-21T21:25:11.581 INFO:teuthology.task.internal:Creating test directory... 2024-09-21T21:25:11.582 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T21:25:11.596 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-21T21:25:11.615 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-21T21:25:11.648 INFO:teuthology.task.internal:Creating archive directory... 2024-09-21T21:25:11.648 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T21:25:11.668 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-21T21:25:11.681 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-21T21:25:11.681 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T21:25:11.745 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T21:25:11.755 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T21:25:11.757 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-21T21:25:11.764 INFO:teuthology.task.internal:Configuring sudo... 2024-09-21T21:25:11.764 DEBUG:teuthology.orchestra.run.smithi129:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T21:25:11.823 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-21T21:25:11.833 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-21T21:25:11.833 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T21:25:11.877 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T21:25:11.961 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T21:25:12.009 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T21:25:12.093 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T21:25:12.131 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T21:25:12.131 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T21:25:12.196 DEBUG:teuthology.orchestra.run.smithi129:> sudo service rsyslog restart 2024-09-21T21:25:12.284 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T21:25:12.353 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-21T21:25:12.367 INFO:teuthology.task.internal:Starting timer... 2024-09-21T21:25:12.367 INFO:teuthology.run_tasks:Running task pcp... 2024-09-21T21:25:12.451 INFO:teuthology.run_tasks:Running task selinux... 2024-09-21T21:25:12.476 DEBUG:teuthology.orchestra.run.smithi129:> sudo service auditd rotate 2024-09-21T21:25:12.527 INFO:teuthology.orchestra.run.smithi129.stdout:Rotating logs: 2024-09-21T21:25:12.529 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-21T21:25:12.529 DEBUG:teuthology.orchestra.run.smithi129:> /usr/sbin/getenforce 2024-09-21T21:25:12.552 INFO:teuthology.orchestra.run.smithi129.stdout:Permissive 2024-09-21T21:25:12.552 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi129.front.sepia.ceph.com': 'permissive'} 2024-09-21T21:25:12.553 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-21T21:25:12.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T21:25:12.617 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-21T21:25:12.617 DEBUG:teuthology.orchestra.run.smithi129:> sudo /usr/sbin/setenforce permissive 2024-09-21T21:25:12.714 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-21T21:25:12.724 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-21T21:25:12.726 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-21T21:25:12.830 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-21T21:25:12.840 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-21T21:25:12.840 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi129.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-21T21:28:50.919 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi129.front.sepia.ceph.com')] 2024-09-21T21:28:50.920 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-21T21:28:50.921 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:28:50.998 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-09-21T21:28:51.081 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-21T21:28:51.082 INFO:teuthology.run_tasks:Running task clock... 2024-09-21T21:28:51.091 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-21T21:28:51.091 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T21:28:51.092 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T21:28:51.151 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T21:28:51.166 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T21:28:51.190 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: ntpd: command not found 2024-09-21T21:28:51.201 INFO:teuthology.orchestra.run.smithi129.stdout:506 Cannot talk to daemon 2024-09-21T21:28:51.215 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T21:28:51.230 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T21:28:51.287 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2024-09-21T21:28:51.291 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T21:28:51.291 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2024-09-21T21:28:51.291 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T21:28:51.291 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T21:28:51.291 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T21:28:51.291 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T21:28:51.291 INFO:teuthology.run_tasks:Running task install... 2024-09-21T21:28:51.299 DEBUG:teuthology.task.install:project ceph 2024-09-21T21:28:51.299 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}} 2024-09-21T21:28:51.299 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'} 2024-09-21T21:28:51.300 INFO:teuthology.task.install:Using flavor: crimson 2024-09-21T21:28:51.305 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-21T21:28:51.305 INFO:teuthology.task.install:extra packages: [] 2024-09-21T21:28:51.306 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'tag': None, 'wait_for_package': False} 2024-09-21T21:28:51.306 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T21:28:51.446 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/ 2024-09-21T21:28:51.446 INFO:teuthology.task.install.rpm:Package version is 19.3.0-5092.g62da54b4 2024-09-21T21:28:51.559 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T21:28:51.559 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T21:28:51.559 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T21:28:51.585 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-21T21:28:51.585 DEBUG:teuthology.orchestra.run.smithi129:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/62da54b4b83a96ca9976b7e1d0c8272a564a2208/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T21:28:51.650 DEBUG:teuthology.orchestra.run.smithi129:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T21:28:51.725 DEBUG:teuthology.orchestra.run.smithi129:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T21:28:51.789 INFO:teuthology.orchestra.run.smithi129.stdout:check_obsoletes = 1 2024-09-21T21:28:51.790 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum clean all 2024-09-21T21:28:52.019 INFO:teuthology.orchestra.run.smithi129.stdout:45 files removed 2024-09-21T21:28:52.046 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-radosgw 2024-09-21T21:28:52.596 INFO:teuthology.orchestra.run.smithi129.stdout:ceph packages for x86_64 334 kB/s | 88 kB 00:00 2024-09-21T21:28:52.871 INFO:teuthology.orchestra.run.smithi129.stdout:ceph noarch packages 105 kB/s | 24 kB 00:00 2024-09-21T21:28:53.139 INFO:teuthology.orchestra.run.smithi129.stdout:ceph source packages 9.0 kB/s | 2.1 kB 00:00 2024-09-21T21:28:53.797 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.3 MB 00:00 2024-09-21T21:29:03.123 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - AppStream 2.5 MB/s | 20 MB 00:08 2024-09-21T21:29:07.440 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-21T21:29:08.908 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - Extras packages 90 kB/s | 19 kB 00:00 2024-09-21T21:29:10.147 INFO:teuthology.orchestra.run.smithi129.stdout:Extra Packages for Enterprise Linux 20 MB/s | 23 MB 00:01 2024-09-21T21:29:15.579 INFO:teuthology.orchestra.run.smithi129.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-21T21:29:17.830 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T21:29:17.842 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:29:17.842 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-21T21:29:17.842 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:29:17.842 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T21:29:17.842 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 10 M 2024-09-21T21:29:17.842 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrading: 2024-09-21T21:29:17.842 INFO:teuthology.orchestra.run.smithi129.stdout: librados2 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 3.5 M 2024-09-21T21:29:17.842 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 3.8 M 2024-09-21T21:29:17.843 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-21T21:29:17.843 INFO:teuthology.orchestra.run.smithi129.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T21:29:17.843 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 6.1 M 2024-09-21T21:29:17.843 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 20 M 2024-09-21T21:29:17.843 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-selinux x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 25 k 2024-09-21T21:29:17.843 INFO:teuthology.orchestra.run.smithi129.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T21:29:17.843 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T21:29:17.843 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T21:29:17.843 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 808 k 2024-09-21T21:29:17.843 INFO:teuthology.orchestra.run.smithi129.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-21T21:29:17.843 INFO:teuthology.orchestra.run.smithi129.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T21:29:17.843 INFO:teuthology.orchestra.run.smithi129.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T21:29:17.843 INFO:teuthology.orchestra.run.smithi129.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T21:29:17.843 INFO:teuthology.orchestra.run.smithi129.stdout: libradosstriper1 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 558 k 2024-09-21T21:29:17.844 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T21:29:17.844 INFO:teuthology.orchestra.run.smithi129.stdout: librgw2 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 5.4 M 2024-09-21T21:29:17.844 INFO:teuthology.orchestra.run.smithi129.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T21:29:17.844 INFO:teuthology.orchestra.run.smithi129.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T21:29:17.844 INFO:teuthology.orchestra.run.smithi129.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T21:29:17.844 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 45 k 2024-09-21T21:29:17.844 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 148 k 2024-09-21T21:29:17.844 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 159 k 2024-09-21T21:29:17.844 INFO:teuthology.orchestra.run.smithi129.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T21:29:17.844 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 314 k 2024-09-21T21:29:17.844 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 302 k 2024-09-21T21:29:17.844 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 99 k 2024-09-21T21:29:17.844 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-21T21:29:17.844 INFO:teuthology.orchestra.run.smithi129.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-21T21:29:17.845 INFO:teuthology.orchestra.run.smithi129.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T21:29:17.845 INFO:teuthology.orchestra.run.smithi129.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T21:29:17.845 INFO:teuthology.orchestra.run.smithi129.stdout:Installing weak dependencies: 2024-09-21T21:29:17.845 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-21T21:29:17.845 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:29:17.845 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T21:29:17.845 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:29:17.845 INFO:teuthology.orchestra.run.smithi129.stdout:Install 31 Packages 2024-09-21T21:29:17.845 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrade 2 Packages 2024-09-21T21:29:17.845 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:29:17.846 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 60 M 2024-09-21T21:29:17.846 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T21:29:18.947 INFO:teuthology.orchestra.run.smithi129.stdout:(1/33): ceph-base-19.3.0-5092.g62da54b4.el9.x86 9.5 MB/s | 6.1 MB 00:00 2024-09-21T21:29:19.105 INFO:teuthology.orchestra.run.smithi129.stdout:(2/33): ceph-radosgw-19.3.0-5092.g62da54b4.el9. 13 MB/s | 10 MB 00:00 2024-09-21T21:29:19.130 INFO:teuthology.orchestra.run.smithi129.stdout:(3/33): ceph-selinux-19.3.0-5092.g62da54b4.el9. 137 kB/s | 25 kB 00:00 2024-09-21T21:29:19.164 INFO:teuthology.orchestra.run.smithi129.stdout:(4/33): libcephfs2-19.3.0-5092.g62da54b4.el9.x8 14 MB/s | 808 kB 00:00 2024-09-21T21:29:19.206 INFO:teuthology.orchestra.run.smithi129.stdout:(5/33): libradosstriper1-19.3.0-5092.g62da54b4. 7.2 MB/s | 558 kB 00:00 2024-09-21T21:29:19.265 INFO:teuthology.orchestra.run.smithi129.stdout:(6/33): python3-ceph-argparse-19.3.0-5092.g62da 774 kB/s | 45 kB 00:00 2024-09-21T21:29:19.332 INFO:teuthology.orchestra.run.smithi129.stdout:(7/33): python3-ceph-common-19.3.0-5092.g62da54 2.2 MB/s | 148 kB 00:00 2024-09-21T21:29:19.524 INFO:teuthology.orchestra.run.smithi129.stdout:(8/33): ceph-common-19.3.0-5092.g62da54b4.el9.x 16 MB/s | 20 MB 00:01 2024-09-21T21:29:19.549 INFO:teuthology.orchestra.run.smithi129.stdout:(9/33): python3-cephfs-19.3.0-5092.g62da54b4.el 735 kB/s | 159 kB 00:00 2024-09-21T21:29:19.574 INFO:teuthology.orchestra.run.smithi129.stdout:(10/33): python3-rados-19.3.0-5092.g62da54b4.el 6.1 MB/s | 314 kB 00:00 2024-09-21T21:29:19.608 INFO:teuthology.orchestra.run.smithi129.stdout:(11/33): python3-rbd-19.3.0-5092.g62da54b4.el9. 5.0 MB/s | 302 kB 00:00 2024-09-21T21:29:19.633 INFO:teuthology.orchestra.run.smithi129.stdout:(12/33): python3-rgw-19.3.0-5092.g62da54b4.el9. 1.7 MB/s | 99 kB 00:00 2024-09-21T21:29:19.709 INFO:teuthology.orchestra.run.smithi129.stdout:(13/33): librgw2-19.3.0-5092.g62da54b4.el9.x86_ 9.8 MB/s | 5.4 MB 00:00 2024-09-21T21:29:19.842 INFO:teuthology.orchestra.run.smithi129.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 457 kB/s | 107 kB 00:00 2024-09-21T21:29:19.867 INFO:teuthology.orchestra.run.smithi129.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 729 kB/s | 171 kB 00:00 2024-09-21T21:29:19.893 INFO:teuthology.orchestra.run.smithi129.stdout:(16/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 871 kB/s | 160 kB 00:00 2024-09-21T21:29:19.910 INFO:teuthology.orchestra.run.smithi129.stdout:(17/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 668 kB/s | 45 kB 00:00 2024-09-21T21:29:19.952 INFO:teuthology.orchestra.run.smithi129.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-21T21:29:19.977 INFO:teuthology.orchestra.run.smithi129.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2024-09-21T21:29:20.019 INFO:teuthology.orchestra.run.smithi129.stdout:(20/33): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 662 kB 00:00 2024-09-21T21:29:20.036 INFO:teuthology.orchestra.run.smithi129.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 612 kB/s | 36 kB 00:00 2024-09-21T21:29:20.061 INFO:teuthology.orchestra.run.smithi129.stdout:(22/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-09-21T21:29:20.078 INFO:teuthology.orchestra.run.smithi129.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 805 kB/s | 47 kB 00:00 2024-09-21T21:29:20.137 INFO:teuthology.orchestra.run.smithi129.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.0 MB/s | 308 kB 00:00 2024-09-21T21:29:20.154 INFO:teuthology.orchestra.run.smithi129.stdout:(25/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 424 kB/s | 26 kB 00:00 2024-09-21T21:29:20.171 INFO:teuthology.orchestra.run.smithi129.stdout:(26/33): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-09-21T21:29:20.205 INFO:teuthology.orchestra.run.smithi129.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-21T21:29:20.238 INFO:teuthology.orchestra.run.smithi129.stdout:(28/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 12 MB/s | 839 kB 00:00 2024-09-21T21:29:20.264 INFO:teuthology.orchestra.run.smithi129.stdout:(29/33): re2-20211101-20.el9.x86_64.rpm 3.2 MB/s | 191 kB 00:00 2024-09-21T21:29:20.347 INFO:teuthology.orchestra.run.smithi129.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-21T21:29:20.456 INFO:teuthology.orchestra.run.smithi129.stdout:(31/33): librados2-19.3.0-5092.g62da54b4.el9.x8 18 MB/s | 3.5 MB 00:00 2024-09-21T21:29:20.556 INFO:teuthology.orchestra.run.smithi129.stdout:(32/33): libarrow-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 4.4 MB 00:00 2024-09-21T21:29:20.623 INFO:teuthology.orchestra.run.smithi129.stdout:(33/33): librbd1-19.3.0-5092.g62da54b4.el9.x86_ 14 MB/s | 3.8 MB 00:00 2024-09-21T21:29:20.628 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:29:20.628 INFO:teuthology.orchestra.run.smithi129.stdout:Total 22 MB/s | 60 MB 00:02 2024-09-21T21:29:20.736 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T21:29:20.774 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T21:29:20.774 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T21:29:21.439 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T21:29:21.439 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T21:29:22.955 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T21:29:23.495 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-21T21:29:23.550 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 2/35 2024-09-21T21:29:23.643 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 2/35 2024-09-21T21:29:23.740 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 3/35 2024-09-21T21:29:23.769 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 4/35 2024-09-21T21:29:23.815 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 4/35 2024-09-21T21:29:23.861 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/35 2024-09-21T21:29:23.899 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/35 2024-09-21T21:29:23.924 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/35 2024-09-21T21:29:23.955 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-21T21:29:23.978 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-21T21:29:24.174 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-21T21:29:24.205 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-21T21:29:24.240 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/35 2024-09-21T21:29:24.283 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ceph-argparse-2:19.3.0-5092.g62da54b4.el9. 11/35 2024-09-21T21:29:24.314 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 12/35 2024-09-21T21:29:24.404 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 13/35 2024-09-21T21:29:24.444 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_6 14/35 2024-09-21T21:29:24.516 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_6 14/35 2024-09-21T21:29:24.556 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : thrift-0.15.0-2.el9.x86_64 15/35 2024-09-21T21:29:24.671 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : re2-1:20211101-20.el9.x86_64 16/35 2024-09-21T21:29:24.711 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 17/35 2024-09-21T21:29:24.766 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 18/35 2024-09-21T21:29:24.801 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 19/35 2024-09-21T21:29:24.830 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 20/35 2024-09-21T21:29:24.867 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 21/35 2024-09-21T21:29:25.259 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-21T21:29:25.309 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 23/35 2024-09-21T21:29:25.409 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 23/35 2024-09-21T21:29:25.440 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 24/35 2024-09-21T21:29:25.476 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-21T21:29:26.047 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-21T21:29:26.079 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 27/35 2024-09-21T21:29:26.150 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 27/35 2024-09-21T21:29:26.232 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 28/35 2024-09-21T21:29:26.265 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ceph-common-2:19.3.0-5092.g62da54b4.el9.x8 29/35 2024-09-21T21:29:28.748 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 30/35 2024-09-21T21:29:28.787 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 30/35 2024-09-21T21:29:29.484 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 30/35 2024-09-21T21:29:29.524 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 31/35 2024-09-21T21:29:29.612 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 31/35 2024-09-21T21:29:29.612 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T21:29:29.612 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T21:29:29.613 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:29:29.655 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 32/35 2024-09-21T21:29:39.986 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 32/35 2024-09-21T21:29:39.986 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /sys 2024-09-21T21:29:39.986 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /proc 2024-09-21T21:29:39.986 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /mnt 2024-09-21T21:29:39.986 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /var/tmp 2024-09-21T21:29:39.986 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /home 2024-09-21T21:29:39.987 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /root 2024-09-21T21:29:39.987 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /tmp 2024-09-21T21:29:39.987 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:29:41.133 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 33/35 2024-09-21T21:29:41.159 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 33/35 2024-09-21T21:29:41.159 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:29:41.159 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T21:29:41.159 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T21:29:41.159 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T21:29:41.159 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:29:41.161 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-21T21:29:41.244 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-21T21:29:41.244 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T21:29:42.577 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T21:29:42.577 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 1/35 2024-09-21T21:29:42.577 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 2/35 2024-09-21T21:29:42.577 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 3/35 2024-09-21T21:29:42.577 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 4/35 2024-09-21T21:29:42.577 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 5/35 2024-09-21T21:29:42.577 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_6 6/35 2024-09-21T21:29:42.577 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 7/35 2024-09-21T21:29:42.578 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ceph-argparse-2:19.3.0-5092.g62da54b4.el9. 8/35 2024-09-21T21:29:42.578 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ceph-common-2:19.3.0-5092.g62da54b4.el9.x8 9/35 2024-09-21T21:29:42.578 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 10/35 2024-09-21T21:29:42.578 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 11/35 2024-09-21T21:29:42.578 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 12/35 2024-09-21T21:29:42.578 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 13/35 2024-09-21T21:29:42.578 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-21T21:29:42.578 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-21T21:29:42.578 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-21T21:29:42.578 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-21T21:29:42.578 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-21T21:29:42.578 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-21T21:29:42.578 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-21T21:29:42.579 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-21T21:29:42.579 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-21T21:29:42.579 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-21T21:29:42.579 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-21T21:29:42.579 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-21T21:29:42.579 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-21T21:29:42.579 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-21T21:29:42.579 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-21T21:29:42.579 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-21T21:29:42.579 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-21T21:29:42.579 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-21T21:29:42.579 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 32/35 2024-09-21T21:29:42.579 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-21T21:29:42.580 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 34/35 2024-09-21T21:29:43.577 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T21:29:43.577 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:29:43.577 INFO:teuthology.orchestra.run.smithi129.stdout:Upgraded: 2024-09-21T21:29:43.577 INFO:teuthology.orchestra.run.smithi129.stdout: librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:43.577 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:43.578 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T21:29:43.578 INFO:teuthology.orchestra.run.smithi129.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T21:29:43.578 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:43.578 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:43.578 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:43.578 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:43.578 INFO:teuthology.orchestra.run.smithi129.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T21:29:43.579 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T21:29:43.579 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T21:29:43.579 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:43.579 INFO:teuthology.orchestra.run.smithi129.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-21T21:29:43.579 INFO:teuthology.orchestra.run.smithi129.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T21:29:43.579 INFO:teuthology.orchestra.run.smithi129.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T21:29:43.579 INFO:teuthology.orchestra.run.smithi129.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T21:29:43.579 INFO:teuthology.orchestra.run.smithi129.stdout: libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:43.579 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T21:29:43.580 INFO:teuthology.orchestra.run.smithi129.stdout: librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:43.580 INFO:teuthology.orchestra.run.smithi129.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T21:29:43.580 INFO:teuthology.orchestra.run.smithi129.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T21:29:43.580 INFO:teuthology.orchestra.run.smithi129.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T21:29:43.580 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:43.580 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:43.580 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:43.580 INFO:teuthology.orchestra.run.smithi129.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T21:29:43.581 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:43.581 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:43.581 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:43.581 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-21T21:29:43.581 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-21T21:29:43.581 INFO:teuthology.orchestra.run.smithi129.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-21T21:29:43.581 INFO:teuthology.orchestra.run.smithi129.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T21:29:43.581 INFO:teuthology.orchestra.run.smithi129.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T21:29:43.581 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:29:43.582 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T21:29:43.939 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-test 2024-09-21T21:29:44.526 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:29 ago on Sat 21 Sep 2024 09:29:15 PM UTC. 2024-09-21T21:29:44.656 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T21:29:44.657 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:29:44.657 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-21T21:29:44.657 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:29:44.658 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T21:29:44.658 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 81 M 2024-09-21T21:29:44.658 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-21T21:29:44.658 INFO:teuthology.orchestra.run.smithi129.stdout: libasan8 x86_64 14.2.1-1.2.el9 appstream 513 k 2024-09-21T21:29:44.658 INFO:teuthology.orchestra.run.smithi129.stdout: libcephsqlite x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 175 k 2024-09-21T21:29:44.658 INFO:teuthology.orchestra.run.smithi129.stdout: libubsan x86_64 11.5.0-2.el9 appstream 183 k 2024-09-21T21:29:44.658 INFO:teuthology.orchestra.run.smithi129.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-09-21T21:29:44.659 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-21T21:29:44.659 INFO:teuthology.orchestra.run.smithi129.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T21:29:44.659 INFO:teuthology.orchestra.run.smithi129.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T21:29:44.659 INFO:teuthology.orchestra.run.smithi129.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-09-21T21:29:44.659 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:29:44.659 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T21:29:44.659 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:29:44.659 INFO:teuthology.orchestra.run.smithi129.stdout:Install 9 Packages 2024-09-21T21:29:44.659 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:29:44.660 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 83 M 2024-09-21T21:29:44.660 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 775 M 2024-09-21T21:29:44.661 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T21:29:45.209 INFO:teuthology.orchestra.run.smithi129.stdout:(1/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 819 kB/s | 94 kB 00:00 2024-09-21T21:29:45.325 INFO:teuthology.orchestra.run.smithi129.stdout:(2/9): libcephsqlite-19.3.0-5092.g62da54b4.el9. 754 kB/s | 175 kB 00:00 2024-09-21T21:29:46.034 INFO:teuthology.orchestra.run.smithi129.stdout:(3/9): libasan8-14.2.1-1.2.el9.x86_64.rpm 622 kB/s | 513 kB 00:00 2024-09-21T21:29:46.060 INFO:teuthology.orchestra.run.smithi129.stdout:(4/9): libubsan-11.5.0-2.el9.x86_64.rpm 249 kB/s | 183 kB 00:00 2024-09-21T21:29:47.419 INFO:teuthology.orchestra.run.smithi129.stdout:(5/9): ceph-test-19.3.0-5092.g62da54b4.el9.x86_ 35 MB/s | 81 MB 00:02 2024-09-21T21:29:47.443 INFO:teuthology.orchestra.run.smithi129.stdout:(6/9): socat-1.7.4.1-6.el9.x86_64.rpm 220 kB/s | 304 kB 00:01 2024-09-21T21:29:47.502 INFO:teuthology.orchestra.run.smithi129.stdout:(7/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 2.0 MB/s | 122 kB 00:00 2024-09-21T21:29:47.605 INFO:teuthology.orchestra.run.smithi129.stdout:(8/9): protobuf-3.14.0-13.el9.x86_64.rpm 659 kB/s | 1.0 MB 00:01 2024-09-21T21:29:48.069 INFO:teuthology.orchestra.run.smithi129.stdout:(9/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 98 kB/s | 64 kB 00:00 2024-09-21T21:29:48.073 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:29:48.073 INFO:teuthology.orchestra.run.smithi129.stdout:Total 24 MB/s | 83 MB 00:03 2024-09-21T21:29:48.157 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T21:29:48.180 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T21:29:48.180 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T21:29:48.779 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T21:29:48.780 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T21:29:49.665 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T21:29:49.732 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-09-21T21:29:49.789 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-09-21T21:29:49.841 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/9 2024-09-21T21:29:49.871 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-09-21T21:29:49.907 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-21T21:29:49.932 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-21T21:29:50.001 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-21T21:29:50.040 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-09-21T21:29:50.075 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 8/9 2024-09-21T21:30:00.881 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 8/9 2024-09-21T21:30:00.966 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 9/9 2024-09-21T21:30:06.960 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 9/9 2024-09-21T21:30:06.961 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 1/9 2024-09-21T21:30:06.961 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 2/9 2024-09-21T21:30:06.961 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-09-21T21:30:06.961 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libasan8-14.2.1-1.2.el9.x86_64 4/9 2024-09-21T21:30:06.961 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-21T21:30:06.962 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-09-21T21:30:06.962 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 7/9 2024-09-21T21:30:06.962 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-09-21T21:30:07.374 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-09-21T21:30:07.374 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:30:07.374 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T21:30:07.374 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:30:07.374 INFO:teuthology.orchestra.run.smithi129.stdout: libasan8-14.2.1-1.2.el9.x86_64 2024-09-21T21:30:07.374 INFO:teuthology.orchestra.run.smithi129.stdout: libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:30:07.374 INFO:teuthology.orchestra.run.smithi129.stdout: libubsan-11.5.0-2.el9.x86_64 2024-09-21T21:30:07.374 INFO:teuthology.orchestra.run.smithi129.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-09-21T21:30:07.374 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-21T21:30:07.374 INFO:teuthology.orchestra.run.smithi129.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T21:30:07.374 INFO:teuthology.orchestra.run.smithi129.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T21:30:07.375 INFO:teuthology.orchestra.run.smithi129.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-09-21T21:30:07.375 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:30:07.375 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T21:30:07.717 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph 2024-09-21T21:30:08.303 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:53 ago on Sat 21 Sep 2024 09:29:15 PM UTC. 2024-09-21T21:30:08.440 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T21:30:08.443 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:30:08.443 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-21T21:30:08.443 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:30:08.443 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T21:30:08.443 INFO:teuthology.orchestra.run.smithi129.stdout: ceph x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 6.5 k 2024-09-21T21:30:08.443 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-21T21:30:08.443 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 2.2 M 2024-09-21T21:30:08.443 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 1.6 M 2024-09-21T21:30:08.444 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 292 k 2024-09-21T21:30:08.444 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 5.1 M 2024-09-21T21:30:08.444 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 70 M 2024-09-21T21:30:08.444 INFO:teuthology.orchestra.run.smithi129.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-21T21:30:08.444 INFO:teuthology.orchestra.run.smithi129.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-21T21:30:08.444 INFO:teuthology.orchestra.run.smithi129.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-21T21:30:08.444 INFO:teuthology.orchestra.run.smithi129.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T21:30:08.444 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T21:30:08.444 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T21:30:08.444 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T21:30:08.444 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T21:30:08.445 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T21:30:08.445 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T21:30:08.445 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-collections 2024-09-21T21:30:08.445 INFO:teuthology.orchestra.run.smithi129.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-21T21:30:08.445 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T21:30:08.445 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T21:30:08.445 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T21:30:08.445 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T21:30:08.445 INFO:teuthology.orchestra.run.smithi129.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T21:30:08.445 INFO:teuthology.orchestra.run.smithi129.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T21:30:08.445 INFO:teuthology.orchestra.run.smithi129.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T21:30:08.445 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T21:30:08.445 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T21:30:08.445 INFO:teuthology.orchestra.run.smithi129.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T21:30:08.446 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T21:30:08.446 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T21:30:08.446 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T21:30:08.446 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T21:30:08.446 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T21:30:08.446 INFO:teuthology.orchestra.run.smithi129.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T21:30:08.446 INFO:teuthology.orchestra.run.smithi129.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T21:30:08.446 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T21:30:08.446 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T21:30:08.446 INFO:teuthology.orchestra.run.smithi129.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T21:30:08.446 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:30:08.446 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T21:30:08.446 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:30:08.446 INFO:teuthology.orchestra.run.smithi129.stdout:Install 37 Packages 2024-09-21T21:30:08.447 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:30:08.447 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 84 M 2024-09-21T21:30:08.447 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 896 M 2024-09-21T21:30:08.447 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T21:30:09.439 INFO:teuthology.orchestra.run.smithi129.stdout:(1/37): ceph-19.3.0-5092.g62da54b4.el9.x86_64.r 48 kB/s | 6.5 kB 00:00 2024-09-21T21:30:09.622 INFO:teuthology.orchestra.run.smithi129.stdout:(2/37): ceph-mgr-19.3.0-5092.g62da54b4.el9.x86_ 5.0 MB/s | 1.6 MB 00:00 2024-09-21T21:30:09.664 INFO:teuthology.orchestra.run.smithi129.stdout:(3/37): ceph-mds-19.3.0-5092.g62da54b4.el9.x86_ 6.1 MB/s | 2.2 MB 00:00 2024-09-21T21:30:09.765 INFO:teuthology.orchestra.run.smithi129.stdout:(4/37): ceph-mgr-modules-core-19.3.0-5092.g62da 2.8 MB/s | 292 kB 00:00 2024-09-21T21:30:09.840 INFO:teuthology.orchestra.run.smithi129.stdout:(5/37): ceph-mon-19.3.0-5092.g62da54b4.el9.x86_ 13 MB/s | 5.1 MB 00:00 2024-09-21T21:30:10.007 INFO:teuthology.orchestra.run.smithi129.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 253 kB 00:00 2024-09-21T21:30:10.057 INFO:teuthology.orchestra.run.smithi129.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-21T21:30:10.116 INFO:teuthology.orchestra.run.smithi129.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 2.3 MB/s | 135 kB 00:00 2024-09-21T21:30:10.170 INFO:teuthology.orchestra.run.smithi129.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 3.7 MB/s | 1.2 MB 00:00 2024-09-21T21:30:10.192 INFO:teuthology.orchestra.run.smithi129.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 1.6 MB/s | 125 kB 00:00 2024-09-21T21:30:10.317 INFO:teuthology.orchestra.run.smithi129.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.4 MB/s | 215 kB 00:00 2024-09-21T21:30:10.351 INFO:teuthology.orchestra.run.smithi129.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-09-21T21:30:10.393 INFO:teuthology.orchestra.run.smithi129.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 840 kB/s | 35 kB 00:00 2024-09-21T21:30:10.451 INFO:teuthology.orchestra.run.smithi129.stdout:(14/37): python3-packaging-20.9-5.el9.noarch.rp 1.3 MB/s | 77 kB 00:00 2024-09-21T21:30:10.485 INFO:teuthology.orchestra.run.smithi129.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-09-21T21:30:10.502 INFO:teuthology.orchestra.run.smithi129.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-21T21:30:10.527 INFO:teuthology.orchestra.run.smithi129.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 994 kB/s | 42 kB 00:00 2024-09-21T21:30:10.586 INFO:teuthology.orchestra.run.smithi129.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-09-21T21:30:10.603 INFO:teuthology.orchestra.run.smithi129.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 2.6 MB/s | 43 kB 00:00 2024-09-21T21:30:10.628 INFO:teuthology.orchestra.run.smithi129.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 177 kB/s | 22 kB 00:00 2024-09-21T21:30:10.670 INFO:teuthology.orchestra.run.smithi129.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-09-21T21:30:10.696 INFO:teuthology.orchestra.run.smithi129.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 461 kB/s | 11 kB 00:00 2024-09-21T21:30:10.721 INFO:teuthology.orchestra.run.smithi129.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 713 kB/s | 18 kB 00:00 2024-09-21T21:30:10.746 INFO:teuthology.orchestra.run.smithi129.stdout:(24/37): python3-cherrypy-18.6.1-2.el9.noarch.r 3.0 MB/s | 358 kB 00:00 2024-09-21T21:30:10.771 INFO:teuthology.orchestra.run.smithi129.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-09-21T21:30:10.797 INFO:teuthology.orchestra.run.smithi129.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 784 kB/s | 19 kB 00:00 2024-09-21T21:30:10.822 INFO:teuthology.orchestra.run.smithi129.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-09-21T21:30:10.847 INFO:teuthology.orchestra.run.smithi129.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 922 kB/s | 46 kB 00:00 2024-09-21T21:30:10.873 INFO:teuthology.orchestra.run.smithi129.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-21T21:30:10.898 INFO:teuthology.orchestra.run.smithi129.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 660 kB/s | 16 kB 00:00 2024-09-21T21:30:10.932 INFO:teuthology.orchestra.run.smithi129.stdout:(31/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.6 MB/s | 90 kB 00:00 2024-09-21T21:30:10.957 INFO:teuthology.orchestra.run.smithi129.stdout:(32/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-09-21T21:30:10.982 INFO:teuthology.orchestra.run.smithi129.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 710 kB/s | 36 kB 00:00 2024-09-21T21:30:11.016 INFO:teuthology.orchestra.run.smithi129.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-09-21T21:30:11.041 INFO:teuthology.orchestra.run.smithi129.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 804 kB/s | 20 kB 00:00 2024-09-21T21:30:11.067 INFO:teuthology.orchestra.run.smithi129.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-09-21T21:30:12.001 INFO:teuthology.orchestra.run.smithi129.stdout:(37/37): ceph-osd-19.3.0-5092.g62da54b4.el9.x86 30 MB/s | 70 MB 00:02 2024-09-21T21:30:12.005 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:30:12.005 INFO:teuthology.orchestra.run.smithi129.stdout:Total 24 MB/s | 84 MB 00:03 2024-09-21T21:30:12.154 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T21:30:12.195 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T21:30:12.195 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T21:30:12.984 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T21:30:12.984 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T21:30:14.672 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T21:30:14.749 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-21T21:30:14.791 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-21T21:30:14.827 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-21T21:30:14.898 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-21T21:30:14.934 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-21T21:30:14.968 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-21T21:30:15.058 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-21T21:30:15.129 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-21T21:30:15.179 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-21T21:30:15.213 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-21T21:30:15.244 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-21T21:30:15.278 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-21T21:30:15.311 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-21T21:30:15.352 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-21T21:30:15.395 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-21T21:30:15.430 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-21T21:30:15.478 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-21T21:30:15.510 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-21T21:30:15.583 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-21T21:30:15.722 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-21T21:30:15.811 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-21T21:30:15.864 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-21T21:30:15.911 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-21T21:30:15.976 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-21T21:30:16.032 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-21T21:30:16.218 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-21T21:30:16.274 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-21T21:30:16.360 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-21T21:30:16.457 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-21T21:30:16.504 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-21T21:30:16.636 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-21T21:30:16.835 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-5092.g62da54b4.el9. 32/37 2024-09-21T21:30:16.860 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 33/37 2024-09-21T21:30:16.886 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 33/37 2024-09-21T21:30:16.887 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:30:16.887 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T21:30:16.887 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T21:30:16.887 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T21:30:16.887 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:30:27.443 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 34/37 2024-09-21T21:30:27.471 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 34/37 2024-09-21T21:30:27.471 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:30:27.471 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T21:30:27.471 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T21:30:27.471 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T21:30:27.471 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:30:28.019 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 35/37 2024-09-21T21:30:28.043 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 35/37 2024-09-21T21:30:28.043 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:30:28.043 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T21:30:28.043 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T21:30:28.043 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T21:30:28.044 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:30:28.287 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 36/37 2024-09-21T21:30:28.310 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 36/37 2024-09-21T21:30:28.311 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:30:28.311 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T21:30:28.311 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T21:30:28.311 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T21:30:28.311 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:30:28.384 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 37/37 2024-09-21T21:30:35.049 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 37/37 2024-09-21T21:30:35.049 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 1/37 2024-09-21T21:30:35.049 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 2/37 2024-09-21T21:30:35.049 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 3/37 2024-09-21T21:30:35.049 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 4/37 2024-09-21T21:30:35.049 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 5/37 2024-09-21T21:30:35.049 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-5092.g62da54b4.el9. 6/37 2024-09-21T21:30:35.049 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-21T21:30:35.049 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-21T21:30:35.049 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-21T21:30:35.050 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-21T21:30:35.050 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-21T21:30:35.050 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-21T21:30:35.050 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-21T21:30:35.050 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-21T21:30:35.050 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-21T21:30:35.050 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-21T21:30:35.050 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-21T21:30:35.050 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-21T21:30:35.050 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-21T21:30:35.050 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-21T21:30:35.050 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-21T21:30:35.050 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-21T21:30:35.051 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-21T21:30:35.051 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-21T21:30:35.051 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-21T21:30:35.052 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-21T21:30:35.052 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-21T21:30:35.052 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-21T21:30:35.052 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-21T21:30:35.052 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-21T21:30:35.052 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-21T21:30:35.052 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-21T21:30:35.052 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-21T21:30:35.053 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-21T21:30:35.053 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-21T21:30:35.053 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-21T21:30:35.917 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-21T21:30:35.917 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:30:35.917 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T21:30:35.917 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:30:35.917 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:30:35.917 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:30:35.917 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:30:35.918 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:30:35.918 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:30:35.918 INFO:teuthology.orchestra.run.smithi129.stdout: lua-5.4.4-4.el9.x86_64 2024-09-21T21:30:35.918 INFO:teuthology.orchestra.run.smithi129.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-21T21:30:35.918 INFO:teuthology.orchestra.run.smithi129.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-21T21:30:35.918 INFO:teuthology.orchestra.run.smithi129.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T21:30:35.918 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T21:30:35.918 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T21:30:35.918 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T21:30:35.918 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T21:30:35.918 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T21:30:35.918 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T21:30:35.918 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T21:30:35.919 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T21:30:35.919 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T21:30:35.919 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T21:30:35.919 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T21:30:35.919 INFO:teuthology.orchestra.run.smithi129.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T21:30:35.919 INFO:teuthology.orchestra.run.smithi129.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T21:30:35.919 INFO:teuthology.orchestra.run.smithi129.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T21:30:35.919 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T21:30:35.919 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T21:30:35.919 INFO:teuthology.orchestra.run.smithi129.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T21:30:35.919 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T21:30:35.919 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T21:30:35.919 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T21:30:35.919 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T21:30:35.920 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T21:30:35.920 INFO:teuthology.orchestra.run.smithi129.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T21:30:35.920 INFO:teuthology.orchestra.run.smithi129.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T21:30:35.920 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T21:30:35.920 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T21:30:35.920 INFO:teuthology.orchestra.run.smithi129.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T21:30:35.920 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:30:35.920 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T21:30:36.301 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-base 2024-09-21T21:30:36.908 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:21 ago on Sat 21 Sep 2024 09:29:15 PM UTC. 2024-09-21T21:30:37.001 INFO:teuthology.orchestra.run.smithi129.stdout:Package ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:30:37.045 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T21:30:37.046 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-09-21T21:30:37.046 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T21:30:37.111 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install cephadm 2024-09-21T21:30:37.709 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:22 ago on Sat 21 Sep 2024 09:29:15 PM UTC. 2024-09-21T21:30:37.839 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T21:30:37.840 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:30:37.840 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-21T21:30:37.840 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:30:37.840 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T21:30:37.840 INFO:teuthology.orchestra.run.smithi129.stdout: cephadm noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 784 k 2024-09-21T21:30:37.840 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:30:37.840 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T21:30:37.840 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:30:37.840 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-09-21T21:30:37.841 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:30:37.841 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 784 k 2024-09-21T21:30:37.842 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 791 k 2024-09-21T21:30:37.842 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T21:30:38.114 INFO:teuthology.orchestra.run.smithi129.stdout:cephadm-19.3.0-5092.g62da54b4.el9.noarch.rpm 2.8 MB/s | 784 kB 00:00 2024-09-21T21:30:38.114 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:30:38.115 INFO:teuthology.orchestra.run.smithi129.stdout:Total 2.8 MB/s | 784 kB 00:00 2024-09-21T21:30:38.115 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T21:30:38.120 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T21:30:38.120 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T21:30:38.126 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T21:30:38.126 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T21:30:38.200 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T21:30:38.710 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-21T21:30:38.827 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-21T21:30:39.238 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-21T21:30:39.469 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-21T21:30:39.470 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:30:39.470 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T21:30:39.470 INFO:teuthology.orchestra.run.smithi129.stdout: cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:30:39.470 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:30:39.470 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T21:30:39.710 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-immutable-object-cache 2024-09-21T21:30:40.300 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:25 ago on Sat 21 Sep 2024 09:29:15 PM UTC. 2024-09-21T21:30:40.435 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T21:30:40.435 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:30:40.435 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2024-09-21T21:30:40.435 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:30:40.436 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T21:30:40.436 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 158 k 2024-09-21T21:30:40.436 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:30:40.436 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T21:30:40.436 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:30:40.436 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-09-21T21:30:40.436 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:30:40.436 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 158 k 2024-09-21T21:30:40.436 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 644 k 2024-09-21T21:30:40.436 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T21:30:40.671 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-immutable-object-cache-19.3.0-5092.g62da54 671 kB/s | 158 kB 00:00 2024-09-21T21:30:40.672 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:30:40.672 INFO:teuthology.orchestra.run.smithi129.stdout:Total 667 kB/s | 158 kB 00:00 2024-09-21T21:30:40.673 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T21:30:40.682 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T21:30:40.682 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T21:30:40.740 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T21:30:40.740 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T21:30:40.888 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T21:30:40.947 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4. 1/1 2024-09-21T21:30:40.972 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4. 1/1 2024-09-21T21:30:40.972 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:30:40.972 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T21:30:40.972 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:30:41.906 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4. 1/1 2024-09-21T21:30:41.906 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:30:41.907 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T21:30:41.907 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:30:41.907 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:30:41.907 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T21:30:42.192 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr 2024-09-21T21:30:42.788 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:27 ago on Sat 21 Sep 2024 09:29:15 PM UTC. 2024-09-21T21:30:42.878 INFO:teuthology.orchestra.run.smithi129.stdout:Package ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:30:42.920 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T21:30:42.921 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-09-21T21:30:42.921 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T21:30:43.013 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr-dashboard 2024-09-21T21:30:43.597 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:28 ago on Sat 21 Sep 2024 09:29:15 PM UTC. 2024-09-21T21:30:43.732 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T21:30:43.733 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:30:43.734 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-21T21:30:43.734 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:30:43.734 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T21:30:43.734 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 14 M 2024-09-21T21:30:43.734 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-21T21:30:43.734 INFO:teuthology.orchestra.run.smithi129.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-21T21:30:43.734 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-grafana-dashboards noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 34 k 2024-09-21T21:30:43.734 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-prometheus-alerts noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 16 k 2024-09-21T21:30:43.734 INFO:teuthology.orchestra.run.smithi129.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-21T21:30:43.734 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-21T21:30:43.734 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-21T21:30:43.734 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-21T21:30:43.735 INFO:teuthology.orchestra.run.smithi129.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-21T21:30:43.735 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T21:30:43.735 INFO:teuthology.orchestra.run.smithi129.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T21:30:43.735 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:30:43.735 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T21:30:43.735 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:30:43.735 INFO:teuthology.orchestra.run.smithi129.stdout:Install 11 Packages 2024-09-21T21:30:43.735 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:30:43.736 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 18 M 2024-09-21T21:30:43.736 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 156 M 2024-09-21T21:30:43.736 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T21:30:44.559 INFO:teuthology.orchestra.run.smithi129.stdout:(1/11): ceph-grafana-dashboards-19.3.0-5092.g62 217 kB/s | 34 kB 00:00 2024-09-21T21:30:44.585 INFO:teuthology.orchestra.run.smithi129.stdout:(2/11): ceph-prometheus-alerts-19.3.0-5092.g62d 90 kB/s | 16 kB 00:00 2024-09-21T21:30:44.710 INFO:teuthology.orchestra.run.smithi129.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.8 MB/s | 269 kB 00:00 2024-09-21T21:30:44.769 INFO:teuthology.orchestra.run.smithi129.stdout:(4/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.2 MB/s | 548 kB 00:00 2024-09-21T21:30:44.794 INFO:teuthology.orchestra.run.smithi129.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 779 kB/s | 19 kB 00:00 2024-09-21T21:30:44.903 INFO:teuthology.orchestra.run.smithi129.stdout:(6/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 19 MB/s | 2.0 MB 00:00 2024-09-21T21:30:44.928 INFO:teuthology.orchestra.run.smithi129.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.7 MB/s | 144 kB 00:00 2024-09-21T21:30:44.953 INFO:teuthology.orchestra.run.smithi129.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-21T21:30:44.979 INFO:teuthology.orchestra.run.smithi129.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 7.3 MB/s | 188 kB 00:00 2024-09-21T21:30:45.146 INFO:teuthology.orchestra.run.smithi129.stdout:(10/11): ceph-mgr-dashboard-19.3.0-5092.g62da54 19 MB/s | 14 MB 00:00 2024-09-21T21:30:45.446 INFO:teuthology.orchestra.run.smithi129.stdout:(11/11): protobuf-compiler-3.14.0-13.el9.x86_64 1.0 MB/s | 863 kB 00:00 2024-09-21T21:30:45.449 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:30:45.449 INFO:teuthology.orchestra.run.smithi129.stdout:Total 11 MB/s | 18 MB 00:01 2024-09-21T21:30:45.481 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T21:30:45.500 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T21:30:45.500 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T21:30:45.769 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T21:30:45.770 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T21:30:46.321 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T21:30:46.403 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-09-21T21:30:46.441 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-09-21T21:30:46.541 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-09-21T21:30:46.683 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-09-21T21:30:46.753 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-09-21T21:30:46.838 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-09-21T21:30:46.880 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-09-21T21:30:46.909 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-09-21T21:30:46.941 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-5092.g62da54b4.el9 9/11 2024-09-21T21:30:50.110 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-5092.g62da54b4.el 10/11 2024-09-21T21:30:50.172 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noa 11/11 2024-09-21T21:30:52.107 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noa 11/11 2024-09-21T21:30:52.107 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-5092.g62da54b4.el 1/11 2024-09-21T21:30:52.107 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noa 2/11 2024-09-21T21:30:52.107 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-5092.g62da54b4.el9 3/11 2024-09-21T21:30:52.107 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-09-21T21:30:52.107 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-09-21T21:30:52.107 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-09-21T21:30:52.107 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-09-21T21:30:52.107 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-09-21T21:30:52.108 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-09-21T21:30:52.108 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-09-21T21:30:52.405 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-09-21T21:30:52.405 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:30:52.405 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T21:30:52.405 INFO:teuthology.orchestra.run.smithi129.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-21T21:30:52.405 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-grafana-dashboards-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:30:52.405 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:30:52.405 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-prometheus-alerts-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:30:52.405 INFO:teuthology.orchestra.run.smithi129.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-21T21:30:52.405 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-21T21:30:52.406 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-21T21:30:52.406 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-21T21:30:52.406 INFO:teuthology.orchestra.run.smithi129.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-21T21:30:52.406 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T21:30:52.406 INFO:teuthology.orchestra.run.smithi129.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T21:30:52.406 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:30:52.406 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T21:30:52.672 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-21T21:30:53.248 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:38 ago on Sat 21 Sep 2024 09:29:15 PM UTC. 2024-09-21T21:30:53.380 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T21:30:53.382 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================= 2024-09-21T21:30:53.382 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-21T21:30:53.382 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================= 2024-09-21T21:30:53.382 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T21:30:53.383 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 7.4 M 2024-09-21T21:30:53.383 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-21T21:30:53.383 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T21:30:53.383 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T21:30:53.383 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T21:30:53.383 INFO:teuthology.orchestra.run.smithi129.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T21:30:53.383 INFO:teuthology.orchestra.run.smithi129.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T21:30:53.383 INFO:teuthology.orchestra.run.smithi129.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T21:30:53.383 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T21:30:53.384 INFO:teuthology.orchestra.run.smithi129.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T21:30:53.384 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T21:30:53.384 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T21:30:53.384 INFO:teuthology.orchestra.run.smithi129.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T21:30:53.384 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:30:53.384 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T21:30:53.384 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================= 2024-09-21T21:30:53.384 INFO:teuthology.orchestra.run.smithi129.stdout:Install 12 Packages 2024-09-21T21:30:53.384 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:30:53.385 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 38 M 2024-09-21T21:30:53.385 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 212 M 2024-09-21T21:30:53.385 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T21:30:53.855 INFO:teuthology.orchestra.run.smithi129.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 191 kB 00:00 2024-09-21T21:30:53.896 INFO:teuthology.orchestra.run.smithi129.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.2 MB/s | 801 kB 00:00 2024-09-21T21:30:53.996 INFO:teuthology.orchestra.run.smithi129.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 231 kB/s | 33 kB 00:00 2024-09-21T21:30:54.047 INFO:teuthology.orchestra.run.smithi129.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 365 kB/s | 18 kB 00:00 2024-09-21T21:30:54.114 INFO:teuthology.orchestra.run.smithi129.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 578 kB/s | 38 kB 00:00 2024-09-21T21:30:54.231 INFO:teuthology.orchestra.run.smithi129.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-50 14 MB/s | 7.4 MB 00:00 2024-09-21T21:30:54.372 INFO:teuthology.orchestra.run.smithi129.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 6.3 MB/s | 3.0 MB 00:00 2024-09-21T21:30:54.464 INFO:teuthology.orchestra.run.smithi129.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-09-21T21:30:54.598 INFO:teuthology.orchestra.run.smithi129.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 10 MB/s | 4.9 MB 00:00 2024-09-21T21:30:54.623 INFO:teuthology.orchestra.run.smithi129.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.5 MB/s | 250 kB 00:00 2024-09-21T21:30:54.724 INFO:teuthology.orchestra.run.smithi129.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 15 MB/s | 5.1 MB 00:00 2024-09-21T21:30:55.257 INFO:teuthology.orchestra.run.smithi129.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-09-21T21:30:55.258 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:30:55.258 INFO:teuthology.orchestra.run.smithi129.stdout:Total 20 MB/s | 38 MB 00:01 2024-09-21T21:30:55.615 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T21:30:55.641 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T21:30:55.642 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T21:30:56.067 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T21:30:56.068 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T21:30:56.825 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T21:30:56.904 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-21T21:30:56.947 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-21T21:30:57.045 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-21T21:30:57.082 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-21T21:30:57.266 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-21T21:30:57.289 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-21T21:30:57.357 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-21T21:30:58.005 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-21T21:30:58.120 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-21T21:30:59.356 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T21:31:00.389 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-21T21:31:00.415 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da5 12/12 2024-09-21T21:31:02.975 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da5 12/12 2024-09-21T21:31:02.976 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da5 1/12 2024-09-21T21:31:02.976 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-21T21:31:02.976 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-21T21:31:02.976 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-21T21:31:02.976 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-21T21:31:02.976 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-21T21:31:02.976 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-21T21:31:02.976 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-21T21:31:02.976 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-21T21:31:02.976 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T21:31:02.976 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-21T21:31:03.382 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-21T21:31:03.382 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:03.382 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T21:31:03.382 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:31:03.382 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T21:31:03.382 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T21:31:03.382 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T21:31:03.382 INFO:teuthology.orchestra.run.smithi129.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T21:31:03.382 INFO:teuthology.orchestra.run.smithi129.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T21:31:03.382 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T21:31:03.383 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T21:31:03.383 INFO:teuthology.orchestra.run.smithi129.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T21:31:03.383 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T21:31:03.383 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T21:31:03.383 INFO:teuthology.orchestra.run.smithi129.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T21:31:03.383 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:03.383 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T21:31:03.862 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr-rook 2024-09-21T21:31:04.459 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:49 ago on Sat 21 Sep 2024 09:29:15 PM UTC. 2024-09-21T21:31:04.595 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T21:31:04.595 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:31:04.595 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-21T21:31:04.595 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:31:04.595 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T21:31:04.595 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-rook noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 49 k 2024-09-21T21:31:04.595 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-21T21:31:04.595 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T21:31:04.595 INFO:teuthology.orchestra.run.smithi129.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T21:31:04.595 INFO:teuthology.orchestra.run.smithi129.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T21:31:04.596 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T21:31:04.596 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T21:31:04.596 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T21:31:04.596 INFO:teuthology.orchestra.run.smithi129.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T21:31:04.596 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T21:31:04.596 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T21:31:04.596 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T21:31:04.596 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T21:31:04.596 INFO:teuthology.orchestra.run.smithi129.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T21:31:04.596 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:04.596 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T21:31:04.596 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:31:04.596 INFO:teuthology.orchestra.run.smithi129.stdout:Install 13 Packages 2024-09-21T21:31:04.596 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:04.597 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 2.2 M 2024-09-21T21:31:04.597 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 26 M 2024-09-21T21:31:04.597 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T21:31:04.900 INFO:teuthology.orchestra.run.smithi129.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 209 kB/s | 19 kB 00:00 2024-09-21T21:31:04.925 INFO:teuthology.orchestra.run.smithi129.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 219 kB/s | 26 kB 00:00 2024-09-21T21:31:04.984 INFO:teuthology.orchestra.run.smithi129.stdout:(3/13): ceph-mgr-rook-19.3.0-5092.g62da54b4.el9 280 kB/s | 49 kB 00:00 2024-09-21T21:31:05.010 INFO:teuthology.orchestra.run.smithi129.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-21T21:31:05.026 INFO:teuthology.orchestra.run.smithi129.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 159 kB 00:00 2024-09-21T21:31:05.068 INFO:teuthology.orchestra.run.smithi129.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 770 kB/s | 32 kB 00:00 2024-09-21T21:31:05.094 INFO:teuthology.orchestra.run.smithi129.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 639 kB/s | 54 kB 00:00 2024-09-21T21:31:05.120 INFO:teuthology.orchestra.run.smithi129.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 279 kB/s | 14 kB 00:00 2024-09-21T21:31:05.169 INFO:teuthology.orchestra.run.smithi129.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.5 MB/s | 279 kB 00:00 2024-09-21T21:31:05.203 INFO:teuthology.orchestra.run.smithi129.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.0 MB/s | 220 kB 00:00 2024-09-21T21:31:05.237 INFO:teuthology.orchestra.run.smithi129.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 884 kB/s | 59 kB 00:00 2024-09-21T21:31:05.262 INFO:teuthology.orchestra.run.smithi129.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-09-21T21:31:05.295 INFO:teuthology.orchestra.run.smithi129.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.0 MB/s | 1.0 MB 00:00 2024-09-21T21:31:05.298 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:31:05.298 INFO:teuthology.orchestra.run.smithi129.stdout:Total 3.2 MB/s | 2.2 MB 00:00 2024-09-21T21:31:05.363 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T21:31:05.379 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T21:31:05.379 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T21:31:05.469 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T21:31:05.471 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T21:31:05.979 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T21:31:06.136 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-21T21:31:06.189 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-21T21:31:06.244 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-21T21:31:06.281 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-21T21:31:06.323 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-21T21:31:06.427 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-21T21:31:06.537 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-21T21:31:06.592 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-21T21:31:07.315 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-21T21:31:07.372 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-21T21:31:07.406 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-21T21:31:07.455 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-21T21:31:07.484 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 13/13 2024-09-21T21:31:08.255 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 13/13 2024-09-21T21:31:08.256 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 1/13 2024-09-21T21:31:08.256 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-21T21:31:08.256 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-21T21:31:08.256 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-21T21:31:08.256 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-21T21:31:08.256 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-21T21:31:08.256 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-21T21:31:08.256 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-21T21:31:08.256 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-21T21:31:08.256 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-21T21:31:08.256 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-21T21:31:08.257 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-21T21:31:08.593 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-21T21:31:08.593 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:08.593 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T21:31:08.593 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:31:08.593 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T21:31:08.593 INFO:teuthology.orchestra.run.smithi129.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T21:31:08.593 INFO:teuthology.orchestra.run.smithi129.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T21:31:08.593 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T21:31:08.594 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T21:31:08.594 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T21:31:08.594 INFO:teuthology.orchestra.run.smithi129.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T21:31:08.594 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T21:31:08.594 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T21:31:08.594 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T21:31:08.594 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T21:31:08.594 INFO:teuthology.orchestra.run.smithi129.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T21:31:08.594 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:08.594 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T21:31:08.880 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr-cephadm 2024-09-21T21:31:09.470 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:54 ago on Sat 21 Sep 2024 09:29:15 PM UTC. 2024-09-21T21:31:09.603 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T21:31:09.604 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:31:09.604 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-21T21:31:09.604 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:31:09.604 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T21:31:09.604 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-cephadm noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 154 k 2024-09-21T21:31:09.604 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-21T21:31:09.604 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T21:31:09.604 INFO:teuthology.orchestra.run.smithi129.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T21:31:09.604 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T21:31:09.604 INFO:teuthology.orchestra.run.smithi129.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T21:31:09.604 INFO:teuthology.orchestra.run.smithi129.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T21:31:09.604 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:09.604 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T21:31:09.605 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:31:09.605 INFO:teuthology.orchestra.run.smithi129.stdout:Install 6 Packages 2024-09-21T21:31:09.605 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:09.605 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 7.0 M 2024-09-21T21:31:09.605 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 33 M 2024-09-21T21:31:09.605 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T21:31:10.233 INFO:teuthology.orchestra.run.smithi129.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-09-21T21:31:10.259 INFO:teuthology.orchestra.run.smithi129.stdout:(2/6): ceph-mgr-cephadm-19.3.0-5092.g62da54b4.e 664 kB/s | 154 kB 00:00 2024-09-21T21:31:10.317 INFO:teuthology.orchestra.run.smithi129.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 991 kB/s | 58 kB 00:00 2024-09-21T21:31:10.343 INFO:teuthology.orchestra.run.smithi129.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-21T21:31:10.376 INFO:teuthology.orchestra.run.smithi129.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.8 MB/s | 548 kB 00:00 2024-09-21T21:31:11.176 INFO:teuthology.orchestra.run.smithi129.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 5.2 MB/s | 6.0 MB 00:01 2024-09-21T21:31:11.179 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:31:11.179 INFO:teuthology.orchestra.run.smithi129.stdout:Total 4.5 MB/s | 7.0 MB 00:01 2024-09-21T21:31:11.249 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T21:31:11.261 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T21:31:11.261 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T21:31:11.363 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T21:31:11.363 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T21:31:11.674 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T21:31:11.841 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-21T21:31:11.886 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-21T21:31:12.399 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-21T21:31:12.488 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-21T21:31:12.557 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-21T21:31:12.582 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 6/6 2024-09-21T21:31:13.352 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 6/6 2024-09-21T21:31:13.353 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/6 2024-09-21T21:31:13.353 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-21T21:31:13.353 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-21T21:31:13.353 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-21T21:31:13.353 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-21T21:31:13.627 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-21T21:31:13.628 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:13.628 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T21:31:13.628 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:31:13.628 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T21:31:13.628 INFO:teuthology.orchestra.run.smithi129.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T21:31:13.628 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T21:31:13.628 INFO:teuthology.orchestra.run.smithi129.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T21:31:13.628 INFO:teuthology.orchestra.run.smithi129.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T21:31:13.628 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:13.628 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T21:31:13.901 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-fuse 2024-09-21T21:31:14.486 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:59 ago on Sat 21 Sep 2024 09:29:15 PM UTC. 2024-09-21T21:31:14.620 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T21:31:14.620 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:31:14.620 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-21T21:31:14.620 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:31:14.620 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T21:31:14.621 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 883 k 2024-09-21T21:31:14.621 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-21T21:31:14.621 INFO:teuthology.orchestra.run.smithi129.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T21:31:14.621 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:14.621 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T21:31:14.621 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:31:14.621 INFO:teuthology.orchestra.run.smithi129.stdout:Install 2 Packages 2024-09-21T21:31:14.621 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:14.622 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 963 k 2024-09-21T21:31:14.622 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 4.1 M 2024-09-21T21:31:14.622 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T21:31:15.054 INFO:teuthology.orchestra.run.smithi129.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 514 kB/s | 80 kB 00:00 2024-09-21T21:31:15.196 INFO:teuthology.orchestra.run.smithi129.stdout:(2/2): ceph-fuse-19.3.0-5092.g62da54b4.el9.x86_ 2.9 MB/s | 883 kB 00:00 2024-09-21T21:31:15.196 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:31:15.197 INFO:teuthology.orchestra.run.smithi129.stdout:Total 1.6 MB/s | 963 kB 00:00 2024-09-21T21:31:15.208 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T21:31:15.233 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T21:31:15.233 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T21:31:15.313 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T21:31:15.314 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T21:31:15.502 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T21:31:15.658 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-21T21:31:15.720 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 2/2 2024-09-21T21:31:16.448 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 2/2 2024-09-21T21:31:16.448 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/2 2024-09-21T21:31:16.736 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-21T21:31:16.736 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:16.737 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T21:31:16.737 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-21T21:31:16.737 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:16.737 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T21:31:17.050 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-volume 2024-09-21T21:31:17.646 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:02 ago on Sat 21 Sep 2024 09:29:15 PM UTC. 2024-09-21T21:31:17.786 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T21:31:17.786 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:31:17.787 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-21T21:31:17.787 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:31:17.787 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T21:31:17.787 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-volume noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 285 k 2024-09-21T21:31:17.787 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:17.787 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T21:31:17.787 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:31:17.787 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-09-21T21:31:17.787 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:17.787 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 285 k 2024-09-21T21:31:17.787 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 1.3 M 2024-09-21T21:31:17.787 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T21:31:18.019 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-volume-19.3.0-5092.g62da54b4.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-21T21:31:18.019 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:31:18.020 INFO:teuthology.orchestra.run.smithi129.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-21T21:31:18.020 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T21:31:18.028 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T21:31:18.028 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T21:31:18.055 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T21:31:18.055 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T21:31:18.284 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T21:31:18.354 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-21T21:31:18.371 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-21T21:31:18.371 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:31:18.371 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T21:31:18.371 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:19.358 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-21T21:31:19.358 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:19.358 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T21:31:19.358 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:31:19.358 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:19.358 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T21:31:19.701 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install librados-devel 2024-09-21T21:31:20.299 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:05 ago on Sat 21 Sep 2024 09:29:15 PM UTC. 2024-09-21T21:31:20.432 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T21:31:20.433 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:31:20.433 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2024-09-21T21:31:20.433 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:31:20.433 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T21:31:20.433 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 125 k 2024-09-21T21:31:20.433 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:20.433 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T21:31:20.433 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:31:20.433 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-09-21T21:31:20.433 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:20.434 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 125 k 2024-09-21T21:31:20.434 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 445 k 2024-09-21T21:31:20.434 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T21:31:20.616 INFO:teuthology.orchestra.run.smithi129.stdout:librados-devel-19.3.0-5092.g62da54b4.el9.x86_64 686 kB/s | 125 kB 00:00 2024-09-21T21:31:20.617 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:31:20.617 INFO:teuthology.orchestra.run.smithi129.stdout:Total 680 kB/s | 125 kB 00:00 2024-09-21T21:31:20.617 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T21:31:20.623 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T21:31:20.623 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T21:31:20.687 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T21:31:20.687 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T21:31:20.841 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T21:31:20.972 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:21.431 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:21.681 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:21.682 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:21.682 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T21:31:21.682 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:31:21.682 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:21.682 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T21:31:21.959 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install libcephfs2 2024-09-21T21:31:22.559 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:07 ago on Sat 21 Sep 2024 09:29:15 PM UTC. 2024-09-21T21:31:22.649 INFO:teuthology.orchestra.run.smithi129.stdout:Package libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:31:22.693 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T21:31:22.694 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-09-21T21:31:22.694 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T21:31:22.764 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install libcephfs-devel 2024-09-21T21:31:23.347 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:08 ago on Sat 21 Sep 2024 09:29:15 PM UTC. 2024-09-21T21:31:23.480 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T21:31:23.480 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:31:23.480 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2024-09-21T21:31:23.481 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:31:23.481 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T21:31:23.481 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 32 k 2024-09-21T21:31:23.481 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:23.481 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T21:31:23.481 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:31:23.481 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-09-21T21:31:23.481 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:23.481 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 32 k 2024-09-21T21:31:23.481 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 142 k 2024-09-21T21:31:23.481 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T21:31:23.646 INFO:teuthology.orchestra.run.smithi129.stdout:libcephfs-devel-19.3.0-5092.g62da54b4.el9.x86_6 193 kB/s | 32 kB 00:00 2024-09-21T21:31:23.646 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:31:23.646 INFO:teuthology.orchestra.run.smithi129.stdout:Total 191 kB/s | 32 kB 00:00 2024-09-21T21:31:23.647 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T21:31:23.650 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T21:31:23.650 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T21:31:23.668 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T21:31:23.669 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T21:31:23.757 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T21:31:23.868 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:24.240 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:24.466 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:24.466 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:24.466 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T21:31:24.466 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:31:24.467 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:24.467 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T21:31:24.667 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install librados2 2024-09-21T21:31:25.246 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:10 ago on Sat 21 Sep 2024 09:29:15 PM UTC. 2024-09-21T21:31:25.334 INFO:teuthology.orchestra.run.smithi129.stdout:Package librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:31:25.378 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T21:31:25.379 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-09-21T21:31:25.379 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T21:31:25.454 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install librbd1 2024-09-21T21:31:26.035 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:11 ago on Sat 21 Sep 2024 09:29:15 PM UTC. 2024-09-21T21:31:26.124 INFO:teuthology.orchestra.run.smithi129.stdout:Package librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:31:26.168 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T21:31:26.169 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-09-21T21:31:26.169 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T21:31:26.240 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-rados 2024-09-21T21:31:26.828 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:11 ago on Sat 21 Sep 2024 09:29:15 PM UTC. 2024-09-21T21:31:26.919 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:31:26.962 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T21:31:26.963 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-09-21T21:31:26.963 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T21:31:27.034 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-rgw 2024-09-21T21:31:27.613 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:12 ago on Sat 21 Sep 2024 09:29:15 PM UTC. 2024-09-21T21:31:27.702 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:31:27.745 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T21:31:27.746 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-09-21T21:31:27.747 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T21:31:27.822 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-cephfs 2024-09-21T21:31:28.401 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:13 ago on Sat 21 Sep 2024 09:29:15 PM UTC. 2024-09-21T21:31:28.493 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:31:28.536 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T21:31:28.537 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-09-21T21:31:28.537 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T21:31:28.605 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-rbd 2024-09-21T21:31:29.191 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:14 ago on Sat 21 Sep 2024 09:29:15 PM UTC. 2024-09-21T21:31:29.280 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:31:29.323 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T21:31:29.324 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-09-21T21:31:29.324 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T21:31:29.400 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install rbd-fuse 2024-09-21T21:31:29.977 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:14 ago on Sat 21 Sep 2024 09:29:15 PM UTC. 2024-09-21T21:31:30.110 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T21:31:30.111 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:31:30.111 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2024-09-21T21:31:30.111 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:31:30.111 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T21:31:30.111 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 91 k 2024-09-21T21:31:30.112 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:30.112 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T21:31:30.112 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:31:30.112 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-09-21T21:31:30.112 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:30.112 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 91 k 2024-09-21T21:31:30.112 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 321 k 2024-09-21T21:31:30.112 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T21:31:30.308 INFO:teuthology.orchestra.run.smithi129.stdout:rbd-fuse-19.3.0-5092.g62da54b4.el9.x86_64.rpm 465 kB/s | 91 kB 00:00 2024-09-21T21:31:30.308 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:31:30.308 INFO:teuthology.orchestra.run.smithi129.stdout:Total 461 kB/s | 91 kB 00:00 2024-09-21T21:31:30.309 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T21:31:30.317 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T21:31:30.317 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T21:31:30.380 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T21:31:30.380 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T21:31:30.526 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T21:31:30.630 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:31.021 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:31.270 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:31.270 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:31.270 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T21:31:31.270 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:31:31.270 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:31.270 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T21:31:31.579 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install rbd-mirror 2024-09-21T21:31:32.175 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:17 ago on Sat 21 Sep 2024 09:29:15 PM UTC. 2024-09-21T21:31:32.314 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T21:31:32.315 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:31:32.315 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2024-09-21T21:31:32.315 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:31:32.315 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T21:31:32.316 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 3.5 M 2024-09-21T21:31:32.316 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:32.316 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T21:31:32.316 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:31:32.316 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-09-21T21:31:32.316 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:32.316 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 3.5 M 2024-09-21T21:31:32.316 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 30 M 2024-09-21T21:31:32.316 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T21:31:32.638 INFO:teuthology.orchestra.run.smithi129.stdout:rbd-mirror-19.3.0-5092.g62da54b4.el9.x86_64.rpm 11 MB/s | 3.5 MB 00:00 2024-09-21T21:31:32.638 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:31:32.638 INFO:teuthology.orchestra.run.smithi129.stdout:Total 11 MB/s | 3.5 MB 00:00 2024-09-21T21:31:32.639 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T21:31:32.649 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T21:31:32.650 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T21:31:32.734 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T21:31:32.734 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T21:31:33.317 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T21:31:33.380 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:33.406 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:33.406 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:31:33.406 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T21:31:33.406 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T21:31:33.406 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T21:31:33.406 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:34.632 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:34.632 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:34.632 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T21:31:34.632 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:31:34.632 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:34.632 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T21:31:34.892 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install rbd-nbd 2024-09-21T21:31:35.494 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:20 ago on Sat 21 Sep 2024 09:29:15 PM UTC. 2024-09-21T21:31:35.630 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T21:31:35.631 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:31:35.631 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2024-09-21T21:31:35.631 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:31:35.631 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T21:31:35.631 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 176 k 2024-09-21T21:31:35.631 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:35.631 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T21:31:35.631 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T21:31:35.631 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-09-21T21:31:35.631 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:35.631 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 176 k 2024-09-21T21:31:35.632 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 685 k 2024-09-21T21:31:35.632 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T21:31:35.850 INFO:teuthology.orchestra.run.smithi129.stdout:rbd-nbd-19.3.0-5092.g62da54b4.el9.x86_64.rpm 807 kB/s | 176 kB 00:00 2024-09-21T21:31:35.851 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:31:35.851 INFO:teuthology.orchestra.run.smithi129.stdout:Total 801 kB/s | 176 kB 00:00 2024-09-21T21:31:35.851 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T21:31:35.861 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T21:31:35.861 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T21:31:35.925 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T21:31:35.925 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T21:31:36.079 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T21:31:36.199 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:36.625 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:36.888 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:36.888 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:36.888 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T21:31:36.889 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:31:36.889 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:36.889 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T21:31:37.201 DEBUG:teuthology.parallel:result is None 2024-09-21T21:31:37.202 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T21:31:37.358 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T21:31:37.380 INFO:teuthology.orchestra.run.smithi129.stdout:19.3.0-5092.g62da54b4.el9 2024-09-21T21:31:37.381 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5092.g62da54b4.el9 2024-09-21T21:31:37.381 INFO:teuthology.task.install:The correct ceph version 19.3.0-5092.g62da54b4 is installed. 2024-09-21T21:31:37.382 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-21T21:31:37.383 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T21:31:37.383 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T21:31:37.446 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-21T21:31:37.446 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T21:31:37.447 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T21:31:37.511 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T21:31:37.575 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-21T21:31:37.576 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T21:31:37.576 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T21:31:37.639 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T21:31:37.704 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-21T21:31:37.704 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T21:31:37.704 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T21:31:37.769 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T21:31:37.834 INFO:teuthology.run_tasks:Running task ceph... 2024-09-21T21:31:37.910 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-21T21:31:37.910 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 777 /var/log/ceph 2024-09-21T21:31:37.948 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-21T21:31:37.949 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-21T21:31:38.014 INFO:tasks.ceph:Creating extra log directories... 2024-09-21T21:31:38.014 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-21T21:31:38.085 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-21T21:31:38.085 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-21T21:31:38.086 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914351', 'branch': 'main', 'description': 'crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_lock_and_fence}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7914351', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 101, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 2950, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'sleep_before_teardown': 0, 'subset': '0/1', 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'targets': {'smithi129.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJdvidMRrCtv9KN6zDI2TAzZamUmqkbYF/hKtDscnnAIwTQ34U488rNsfrdxy5tTdzgS7O5dzMaijm3a5/SSBI4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['rbd/test_lock_fence.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-21_20:56:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2902'} 2024-09-21T21:31:38.086 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-21T21:31:38.135 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m0777 -- /var/run/ceph 2024-09-21T21:31:38.200 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T21:31:38.200 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T21:31:38.256 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T21:31:38.256 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_1 2024-09-21T21:31:38.311 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T21:31:38.311 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T21:31:38.311 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-21T21:31:38.311 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T21:31:38.311 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T21:31:38.312 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-09-21 21:31:33.666865336 +0000 2024-09-21T21:31:38.312 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-09-21 21:28:32.792087034 +0000 2024-09-21T21:31:38.312 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-09-21 21:28:32.792087034 +0000 2024-09-21T21:31:38.312 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-09-21 21:28:32.792087034 +0000 2024-09-21T21:31:38.312 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T21:31:38.374 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-09-21T21:31:38.375 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-09-21T21:31:38.375 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000273177 s, 1.9 MB/s 2024-09-21T21:31:38.376 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T21:31:38.433 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_2 2024-09-21T21:31:38.488 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T21:31:38.489 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T21:31:38.489 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-21T21:31:38.489 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T21:31:38.489 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T21:31:38.489 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-09-21 21:31:33.666865336 +0000 2024-09-21T21:31:38.489 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-09-21 21:28:33.113080117 +0000 2024-09-21T21:31:38.489 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-09-21 21:28:33.113080117 +0000 2024-09-21T21:31:38.489 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-09-21 21:28:33.113080117 +0000 2024-09-21T21:31:38.489 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T21:31:38.551 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-09-21T21:31:38.551 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-09-21T21:31:38.551 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000209346 s, 2.4 MB/s 2024-09-21T21:31:38.552 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T21:31:38.608 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_3 2024-09-21T21:31:38.663 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T21:31:38.663 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T21:31:38.664 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-21T21:31:38.664 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T21:31:38.664 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T21:31:38.664 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-09-21 21:31:33.666865336 +0000 2024-09-21T21:31:38.664 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-09-21 21:28:33.441073049 +0000 2024-09-21T21:31:38.664 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-09-21 21:28:33.441073049 +0000 2024-09-21T21:31:38.664 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-09-21 21:28:33.441073049 +0000 2024-09-21T21:31:38.664 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T21:31:38.727 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-09-21T21:31:38.727 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-09-21T21:31:38.727 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.00020935 s, 2.4 MB/s 2024-09-21T21:31:38.728 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T21:31:38.785 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_4 2024-09-21T21:31:38.840 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T21:31:38.840 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T21:31:38.840 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-21T21:31:38.840 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T21:31:38.840 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T21:31:38.840 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-09-21 21:31:33.667865315 +0000 2024-09-21T21:31:38.840 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-09-21 21:28:33.766066046 +0000 2024-09-21T21:31:38.840 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-09-21 21:28:33.766066046 +0000 2024-09-21T21:31:38.841 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-09-21 21:28:33.766066046 +0000 2024-09-21T21:31:38.841 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T21:31:38.904 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-09-21T21:31:38.904 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-09-21T21:31:38.904 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000192174 s, 2.7 MB/s 2024-09-21T21:31:38.905 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T21:31:38.962 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T21:31:38.962 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi129.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-21T21:31:38.962 INFO:tasks.ceph:Generating config... 2024-09-21T21:31:38.963 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-09-21T21:31:38.963 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-21T21:31:38.963 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-21T21:31:38.963 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-21T21:31:38.964 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-09-21T21:31:38.964 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-21T21:31:38.964 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-21T21:31:38.964 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-21T21:31:38.964 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-09-21T21:31:38.964 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-21T21:31:38.964 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-09-21T21:31:38.964 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-09-21T21:31:38.964 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-21T21:31:38.964 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-21T21:31:38.964 INFO:tasks.ceph:[osd] debug seastore = 20 2024-09-21T21:31:38.964 INFO:tasks.ceph:[osd] debug seastore backref = 20 2024-09-21T21:31:38.964 INFO:tasks.ceph:[osd] debug seastore cache = 20 2024-09-21T21:31:38.964 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2024-09-21T21:31:38.965 INFO:tasks.ceph:[osd] debug seastore device = 20 2024-09-21T21:31:38.965 INFO:tasks.ceph:[osd] debug seastore epm = 20 2024-09-21T21:31:38.965 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2024-09-21T21:31:38.965 INFO:tasks.ceph:[osd] debug seastore journal = 20 2024-09-21T21:31:38.965 INFO:tasks.ceph:[osd] debug seastore lba = 20 2024-09-21T21:31:38.965 INFO:tasks.ceph:[osd] debug seastore odata = 20 2024-09-21T21:31:38.965 INFO:tasks.ceph:[osd] debug seastore ompap = 20 2024-09-21T21:31:38.965 INFO:tasks.ceph:[osd] debug seastore onode = 20 2024-09-21T21:31:38.965 INFO:tasks.ceph:[osd] debug seastore t = 20 2024-09-21T21:31:38.965 INFO:tasks.ceph:[osd] debug seastore tm = 20 2024-09-21T21:31:38.965 INFO:tasks.ceph:[osd] osd objectstore = seastore 2024-09-21T21:31:38.965 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-21T21:31:38.965 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-09-21T21:31:38.965 INFO:tasks.ceph:[global] ms client mode = crc 2024-09-21T21:31:38.965 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-09-21T21:31:38.966 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-09-21T21:31:38.966 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-09-21T21:31:38.966 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-09-21T21:31:38.966 INFO:tasks.ceph:[global] ms service mode = crc 2024-09-21T21:31:38.966 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-21T21:31:38.966 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-21T21:31:38.966 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-1 2024-09-21T21:31:38.966 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 2-3 2024-09-21T21:31:38.966 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 4-5 2024-09-21T21:31:38.966 INFO:tasks.ceph:Setting up mon.a... 2024-09-21T21:31:38.966 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-21T21:31:39.068 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/ceph.keyring 2024-09-21T21:31:39.071 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-21T21:31:39.122 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-21T21:31:39.196 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.129')] 2024-09-21T21:31:39.197 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.129', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}} 2024-09-21T21:31:39.198 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T21:31:39.198 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-21T21:31:39.252 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.129 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-21T21:31:39.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:31:39.342+0000 7fd07df1ef00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:31:39.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:31:39.342+0000 7fd07df1ef00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:31:39.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:31:39.343+0000 7fd07df1ef00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:31:39.345 INFO:teuthology.orchestra.run.smithi129.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-21T21:31:39.345 INFO:teuthology.orchestra.run.smithi129.stdout:monmaptool: generated fsid 9c86c2d1-86e4-4d70-bab7-28b6d262ee29 2024-09-21T21:31:39.345 INFO:teuthology.orchestra.run.smithi129.stdout:setting min_mon_release = quincy 2024-09-21T21:31:39.345 INFO:teuthology.orchestra.run.smithi129.stdout:epoch 0 2024-09-21T21:31:39.345 INFO:teuthology.orchestra.run.smithi129.stdout:fsid 9c86c2d1-86e4-4d70-bab7-28b6d262ee29 2024-09-21T21:31:39.345 INFO:teuthology.orchestra.run.smithi129.stdout:last_changed 2024-09-21T21:31:39.344012+0000 2024-09-21T21:31:39.345 INFO:teuthology.orchestra.run.smithi129.stdout:created 2024-09-21T21:31:39.344012+0000 2024-09-21T21:31:39.345 INFO:teuthology.orchestra.run.smithi129.stdout:min_mon_release 17 (quincy) 2024-09-21T21:31:39.345 INFO:teuthology.orchestra.run.smithi129.stdout:election_strategy: 1 2024-09-21T21:31:39.346 INFO:teuthology.orchestra.run.smithi129.stdout:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2024-09-21T21:31:39.346 INFO:teuthology.orchestra.run.smithi129.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-09-21T21:31:39.347 DEBUG:teuthology.orchestra.run.smithi129:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-21T21:31:39.362 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 9c86c2d1-86e4-4d70-bab7-28b6d262ee29... 2024-09-21T21:31:39.363 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-21T21:31:39.445 INFO:teuthology.orchestra.run.smithi129.stdout:[global] 2024-09-21T21:31:39.445 INFO:teuthology.orchestra.run.smithi129.stdout: chdir = "" 2024-09-21T21:31:39.446 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-21T21:31:39.446 INFO:teuthology.orchestra.run.smithi129.stdout: auth supported = cephx 2024-09-21T21:31:39.446 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:39.446 INFO:teuthology.orchestra.run.smithi129.stdout: filestore xattr use omap = true 2024-09-21T21:31:39.446 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:39.446 INFO:teuthology.orchestra.run.smithi129.stdout: mon clock drift allowed = 1.000 2024-09-21T21:31:39.446 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:39.447 INFO:teuthology.orchestra.run.smithi129.stdout: osd crush chooseleaf type = 0 2024-09-21T21:31:39.447 INFO:teuthology.orchestra.run.smithi129.stdout: auth debug = true 2024-09-21T21:31:39.447 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:39.447 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on old message = true 2024-09-21T21:31:39.447 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on bug = true 2024-09-21T21:31:39.447 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:39.448 INFO:teuthology.orchestra.run.smithi129.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-21T21:31:39.448 INFO:teuthology.orchestra.run.smithi129.stdout: mon pg warn max object skew = 0 2024-09-21T21:31:39.448 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:39.448 INFO:teuthology.orchestra.run.smithi129.stdout: # disable pg_autoscaler by default for new pools 2024-09-21T21:31:39.448 INFO:teuthology.orchestra.run.smithi129.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-21T21:31:39.448 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:39.448 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default size = 2 2024-09-21T21:31:39.449 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:39.449 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow primary affinity = true 2024-09-21T21:31:39.449 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow pg remap = true 2024-09-21T21:31:39.449 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on legacy crush tunables = false 2024-09-21T21:31:39.449 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on crush straw calc version zero = false 2024-09-21T21:31:39.449 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on no sortbitwise = false 2024-09-21T21:31:39.449 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on osd down out interval zero = false 2024-09-21T21:31:39.450 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on too few osds = false 2024-09-21T21:31:39.450 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-21T21:31:39.450 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-21T21:31:39.450 INFO:teuthology.orchestra.run.smithi129.stdout: mon_allow_pool_size_one = true 2024-09-21T21:31:39.450 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:39.450 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-21T21:31:39.451 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:39.451 INFO:teuthology.orchestra.run.smithi129.stdout: osd default data pool replay window = 5 2024-09-21T21:31:39.451 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:39.451 INFO:teuthology.orchestra.run.smithi129.stdout: mon allow pool delete = true 2024-09-21T21:31:39.451 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:39.451 INFO:teuthology.orchestra.run.smithi129.stdout: mon cluster log file level = debug 2024-09-21T21:31:39.451 INFO:teuthology.orchestra.run.smithi129.stdout: debug asserts on shutdown = true 2024-09-21T21:31:39.451 INFO:teuthology.orchestra.run.smithi129.stdout: mon health detail to clog = false 2024-09-21T21:31:39.451 INFO:teuthology.orchestra.run.smithi129.stdout: mon host = 172.21.15.129 2024-09-21T21:31:39.452 INFO:teuthology.orchestra.run.smithi129.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-09-21T21:31:39.453 INFO:teuthology.orchestra.run.smithi129.stdout: ms client mode = crc 2024-09-21T21:31:39.453 INFO:teuthology.orchestra.run.smithi129.stdout: ms cluster mode = crc 2024-09-21T21:31:39.453 INFO:teuthology.orchestra.run.smithi129.stdout: ms mon client mode = crc 2024-09-21T21:31:39.454 INFO:teuthology.orchestra.run.smithi129.stdout: ms mon cluster mode = crc 2024-09-21T21:31:39.454 INFO:teuthology.orchestra.run.smithi129.stdout: ms mon service mode = crc 2024-09-21T21:31:39.454 INFO:teuthology.orchestra.run.smithi129.stdout: ms service mode = crc 2024-09-21T21:31:39.454 INFO:teuthology.orchestra.run.smithi129.stdout: fsid = 9c86c2d1-86e4-4d70-bab7-28b6d262ee29 2024-09-21T21:31:39.454 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:39.454 INFO:teuthology.orchestra.run.smithi129.stdout:[osd] 2024-09-21T21:31:39.455 INFO:teuthology.orchestra.run.smithi129.stdout: osd journal size = 100 2024-09-21T21:31:39.455 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:39.455 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub load threshold = 5.0 2024-09-21T21:31:39.455 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub max interval = 600 2024-09-21T21:31:39.455 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock profile = high_recovery_ops 2024-09-21T21:31:39.455 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:39.456 INFO:teuthology.orchestra.run.smithi129.stdout: osd recover clone overlap = true 2024-09-21T21:31:39.456 INFO:teuthology.orchestra.run.smithi129.stdout: osd recovery max chunk = 1048576 2024-09-21T21:31:39.456 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:39.456 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug shutdown = true 2024-09-21T21:31:39.456 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug op order = true 2024-09-21T21:31:39.456 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify stray on activate = true 2024-09-21T21:31:39.456 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug trim objects = true 2024-09-21T21:31:39.457 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:39.457 INFO:teuthology.orchestra.run.smithi129.stdout: osd open classes on start = true 2024-09-21T21:31:39.457 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug pg log writeout = true 2024-09-21T21:31:39.457 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:39.457 INFO:teuthology.orchestra.run.smithi129.stdout: osd deep scrub update digest min age = 30 2024-09-21T21:31:39.457 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:39.457 INFO:teuthology.orchestra.run.smithi129.stdout: osd map max advance = 10 2024-09-21T21:31:39.458 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:39.458 INFO:teuthology.orchestra.run.smithi129.stdout: journal zero on create = true 2024-09-21T21:31:39.458 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:39.458 INFO:teuthology.orchestra.run.smithi129.stdout: filestore ondisk finisher threads = 3 2024-09-21T21:31:39.458 INFO:teuthology.orchestra.run.smithi129.stdout: filestore apply finisher threads = 3 2024-09-21T21:31:39.458 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:39.459 INFO:teuthology.orchestra.run.smithi129.stdout: bdev debug aio = true 2024-09-21T21:31:39.459 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug misdirected ops = true 2024-09-21T21:31:39.459 INFO:teuthology.orchestra.run.smithi129.stdout: debug monc = 20 2024-09-21T21:31:39.459 INFO:teuthology.orchestra.run.smithi129.stdout: crimson alien thread cpu cores = 6-7 2024-09-21T21:31:39.459 INFO:teuthology.orchestra.run.smithi129.stdout: crimson osd obc lru size = 10 2024-09-21T21:31:39.459 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-21T21:31:39.460 INFO:teuthology.orchestra.run.smithi129.stdout: debug osd = 20 2024-09-21T21:31:39.460 INFO:teuthology.orchestra.run.smithi129.stdout: debug seastore = 20 2024-09-21T21:31:39.460 INFO:teuthology.orchestra.run.smithi129.stdout: debug seastore backref = 20 2024-09-21T21:31:39.460 INFO:teuthology.orchestra.run.smithi129.stdout: debug seastore cache = 20 2024-09-21T21:31:39.460 INFO:teuthology.orchestra.run.smithi129.stdout: debug seastore cleaner = 20 2024-09-21T21:31:39.460 INFO:teuthology.orchestra.run.smithi129.stdout: debug seastore device = 20 2024-09-21T21:31:39.460 INFO:teuthology.orchestra.run.smithi129.stdout: debug seastore epm = 20 2024-09-21T21:31:39.461 INFO:teuthology.orchestra.run.smithi129.stdout: debug seastore fixedkv tree = 20 2024-09-21T21:31:39.461 INFO:teuthology.orchestra.run.smithi129.stdout: debug seastore journal = 20 2024-09-21T21:31:39.461 INFO:teuthology.orchestra.run.smithi129.stdout: debug seastore lba = 20 2024-09-21T21:31:39.461 INFO:teuthology.orchestra.run.smithi129.stdout: debug seastore odata = 20 2024-09-21T21:31:39.461 INFO:teuthology.orchestra.run.smithi129.stdout: debug seastore ompap = 20 2024-09-21T21:31:39.461 INFO:teuthology.orchestra.run.smithi129.stdout: debug seastore onode = 20 2024-09-21T21:31:39.462 INFO:teuthology.orchestra.run.smithi129.stdout: debug seastore t = 20 2024-09-21T21:31:39.462 INFO:teuthology.orchestra.run.smithi129.stdout: debug seastore tm = 20 2024-09-21T21:31:39.462 INFO:teuthology.orchestra.run.smithi129.stdout: osd objectstore = seastore 2024-09-21T21:31:39.462 INFO:teuthology.orchestra.run.smithi129.stdout: osd shutdown pgref assert = True 2024-09-21T21:31:39.462 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:39.462 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr] 2024-09-21T21:31:39.463 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-21T21:31:39.463 INFO:teuthology.orchestra.run.smithi129.stdout: debug mgr = 20 2024-09-21T21:31:39.463 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-21T21:31:39.463 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-21T21:31:39.463 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-21T21:31:39.463 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-21T21:31:39.464 INFO:teuthology.orchestra.run.smithi129.stdout: mgr/telemetry/nag = false 2024-09-21T21:31:39.464 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:39.464 INFO:teuthology.orchestra.run.smithi129.stdout:[mon] 2024-09-21T21:31:39.464 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-21T21:31:39.464 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-21T21:31:39.464 INFO:teuthology.orchestra.run.smithi129.stdout: debug paxos = 20 2024-09-21T21:31:39.464 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-21T21:31:39.465 INFO:teuthology.orchestra.run.smithi129.stdout: mon data avail warn = 5 2024-09-21T21:31:39.465 INFO:teuthology.orchestra.run.smithi129.stdout: mon mgr mkfs grace = 240 2024-09-21T21:31:39.465 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-21T21:31:39.465 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd reporter subtree level = osd 2024-09-21T21:31:39.465 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd prime pg temp = true 2024-09-21T21:31:39.465 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-21T21:31:39.466 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:39.466 INFO:teuthology.orchestra.run.smithi129.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-21T21:31:39.466 INFO:teuthology.orchestra.run.smithi129.stdout: auth mon ticket ttl = 660 # 11m 2024-09-21T21:31:39.466 INFO:teuthology.orchestra.run.smithi129.stdout: auth service ticket ttl = 240 # 4m 2024-09-21T21:31:39.466 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:39.466 INFO:teuthology.orchestra.run.smithi129.stdout: # don't complain about insecure global_id in the test suite 2024-09-21T21:31:39.467 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-21T21:31:39.467 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-21T21:31:39.467 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:39.467 INFO:teuthology.orchestra.run.smithi129.stdout: # 1m isn't quite enough 2024-09-21T21:31:39.467 INFO:teuthology.orchestra.run.smithi129.stdout: mon_down_mkfs_grace = 2m 2024-09-21T21:31:39.467 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:39.468 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_filestore_osds = false 2024-09-21T21:31:39.468 INFO:teuthology.orchestra.run.smithi129.stdout: mon min osdmap epochs = 50 2024-09-21T21:31:39.468 INFO:teuthology.orchestra.run.smithi129.stdout: mon osdmap full prune interval = 2 2024-09-21T21:31:39.468 INFO:teuthology.orchestra.run.smithi129.stdout: mon osdmap full prune min = 15 2024-09-21T21:31:39.468 INFO:teuthology.orchestra.run.smithi129.stdout: mon osdmap full prune txsize = 2 2024-09-21T21:31:39.468 INFO:teuthology.orchestra.run.smithi129.stdout: paxos service trim min = 10 2024-09-21T21:31:39.468 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default crimson = True 2024-09-21T21:31:39.469 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:39.469 INFO:teuthology.orchestra.run.smithi129.stdout:[client] 2024-09-21T21:31:39.469 INFO:teuthology.orchestra.run.smithi129.stdout: rgw cache enabled = true 2024-09-21T21:31:39.469 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable ops log = true 2024-09-21T21:31:39.469 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable usage log = true 2024-09-21T21:31:39.469 INFO:teuthology.orchestra.run.smithi129.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-21T21:31:39.470 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-21T21:31:39.470 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.a] 2024-09-21T21:31:39.470 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.0] 2024-09-21T21:31:39.470 INFO:teuthology.orchestra.run.smithi129.stdout: crimson seastar cpu cores = 0-1 2024-09-21T21:31:39.470 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.1] 2024-09-21T21:31:39.470 INFO:teuthology.orchestra.run.smithi129.stdout: crimson seastar cpu cores = 2-3 2024-09-21T21:31:39.470 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.2] 2024-09-21T21:31:39.471 INFO:teuthology.orchestra.run.smithi129.stdout: crimson seastar cpu cores = 4-5 2024-09-21T21:31:39.472 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-21T21:31:39.473 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-21T21:31:39.577 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-21T21:31:39.577 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T21:31:39.577 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-21T21:31:39.592 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T21:31:39.592 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-21T21:31:39.647 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi129.front.sepia.ceph.com 2024-09-21T21:31:39.647 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T21:31:39.648 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-21T21:31:39.648 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-21T21:31:39.722 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T21:31:39.722 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-21T21:31:39.778 INFO:tasks.ceph:Setting up mon nodes... 2024-09-21T21:31:39.779 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-21T21:31:39.779 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-21T21:31:39.878 INFO:teuthology.orchestra.run.smithi129.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-21T21:31:39.881 INFO:tasks.ceph:Setting up mds nodes... 2024-09-21T21:31:39.881 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-21T21:31:39.882 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-21T21:31:39.931 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-21T21:31:39.944 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-21T21:31:39.944 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi129.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-21T21:31:39.945 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-21T21:31:39.969 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T21:31:39.969 INFO:tasks.ceph:role: osd.0 2024-09-21T21:31:39.969 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi129.front.sepia.ceph.com 2024-09-21T21:31:39.970 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-21T21:31:40.041 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T21:31:40.041 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T21:31:40.041 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T21:31:40.041 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T21:31:40.041 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T21:31:40.041 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2024-09-21T21:31:40.042 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T21:31:40.042 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T21:31:40.042 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T21:31:40.042 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T21:31:40.553 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2024-09-21T21:31:40.555 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2024-09-21T21:31:40.556 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-21T21:31:40.679 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-21T21:31:40.711 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-21T21:31:40.776 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T21:31:40.776 INFO:tasks.ceph:role: osd.1 2024-09-21T21:31:40.776 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi129.front.sepia.ceph.com 2024-09-21T21:31:40.776 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-21T21:31:40.849 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T21:31:40.849 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T21:31:40.849 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T21:31:40.849 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T21:31:40.849 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T21:31:40.849 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2024-09-21T21:31:40.849 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T21:31:40.850 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T21:31:40.850 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T21:31:40.850 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T21:31:41.311 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2024-09-21T21:31:41.313 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2024-09-21T21:31:41.313 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-21T21:31:41.404 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-21T21:31:41.433 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-21T21:31:41.498 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T21:31:41.498 INFO:tasks.ceph:role: osd.2 2024-09-21T21:31:41.498 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi129.front.sepia.ceph.com 2024-09-21T21:31:41.498 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-21T21:31:41.570 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T21:31:41.570 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T21:31:41.570 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T21:31:41.570 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T21:31:41.571 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T21:31:41.571 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2024-09-21T21:31:41.571 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T21:31:41.571 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T21:31:41.571 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T21:31:41.571 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T21:31:42.057 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2024-09-21T21:31:42.059 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2024-09-21T21:31:42.060 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-21T21:31:42.212 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-21T21:31:42.243 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T21:31:42.527 INFO:teuthology.orchestra.run.smithi129.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-21T21:31:42.528 INFO:teuthology.orchestra.run.smithi129.stderr:INFO 2024-09-21 21:31:42,528 seastar - Reactor backend: linux-aio 2024-09-21T21:31:42.543 INFO:teuthology.orchestra.run.smithi129.stderr:==27636==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-21T21:31:42.649 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x8b56c 0x10d76c 0xdd31a 0x4ab1c 0x3816dcd 0x5de4092 0x5dfc2ed 0x713a942 0x714c115 0x714d5b2 0x714d637 0x714d739 0x714d831 0x714da3e 0x714dc24 0x714df73 0x714e317 0x714e547 0x714e7b1 0x714ea7e 0x714ec9c 0x714ed54 0x714ee1a 0x714eefb 0x5c46eec 0x5c46ff6 0x5c470b2 0x5c4716e 0x5c4722a 0x5c47377 0x5c47502 0x5c478c3 0x5c616c6 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3961ae8 0x39632c2 0x3803c18 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:42.649 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:42.661 INFO:teuthology.orchestra.run.smithi129.stderr:INFO 2024-09-21 21:31:42,660 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-21T21:31:42.661 INFO:teuthology.orchestra.run.smithi129.stderr:INFO 2024-09-21 21:31:42,660 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 0-1 2024-09-21T21:31:42.662 INFO:teuthology.orchestra.run.smithi129.stderr:DEBUG 2024-09-21 21:31:42,661 [shard 0:main] seastar - reactor::drain 2024-09-21T21:31:43.172 INFO:teuthology.orchestra.run.smithi129.stderr:----------------------------------------------------- 2024-09-21T21:31:43.172 INFO:teuthology.orchestra.run.smithi129.stderr:Suppressions used: 2024-09-21T21:31:43.172 INFO:teuthology.orchestra.run.smithi129.stderr: count bytes template 2024-09-21T21:31:43.172 INFO:teuthology.orchestra.run.smithi129.stderr: 1 8 MallocExtension::Register 2024-09-21T21:31:43.172 INFO:teuthology.orchestra.run.smithi129.stderr:----------------------------------------------------- 2024-09-21T21:31:43.173 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-21T21:31:43.228 INFO:teuthology.orchestra.run.smithi129.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-21T21:31:43.232 INFO:teuthology.orchestra.run.smithi129.stderr:INFO 2024-09-21 21:31:43,231 seastar - Reactor backend: linux-aio 2024-09-21T21:31:43.239 INFO:teuthology.orchestra.run.smithi129.stderr:==27634==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-21T21:31:43.525 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x10797d 0xdd305 0x4b14c 0x3b8bfdd 0x3b8c15e 0xb9d336b 0xb9d6dd9 0xba1a7fc 0xb9bba82 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:43.526 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:43.700 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x8b571 0x10d76c 0xdd31a 0x4b14c 0x60f7453 0x60f75d7 0x60f785f 0xba17baf 0xba17cbb 0xba17ecc 0xba18068 0xba183fe 0xba18610 0xb9b548f 0xb9b58b4 0xb9bb877 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:43.700 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:43.822 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xdd323 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:43.822 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:43.879 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4abd1 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:43.880 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:43.897 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525ec3 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:43.897 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:43.954 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5524e57 0x5525e58 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:43.955 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:43.987 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525162 0x9e16cf9 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:43.987 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:44.062 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 230 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525d25 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:44.062 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:44.517 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x9e16cec 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:44.518 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:44.575 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x10797d 0xdd305 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:44.575 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:45.548 INFO:teuthology.orchestra.run.smithi129.stderr:----------------------------------------------------- 2024-09-21T21:31:45.548 INFO:teuthology.orchestra.run.smithi129.stderr:Suppressions used: 2024-09-21T21:31:45.548 INFO:teuthology.orchestra.run.smithi129.stderr: count bytes template 2024-09-21T21:31:45.548 INFO:teuthology.orchestra.run.smithi129.stderr: 1 8 MallocExtension::Register 2024-09-21T21:31:45.548 INFO:teuthology.orchestra.run.smithi129.stderr:----------------------------------------------------- 2024-09-21T21:31:45.548 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-21T21:31:45.573 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T21:31:45.598 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T21:31:45.886 INFO:teuthology.orchestra.run.smithi129.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-21T21:31:45.887 INFO:teuthology.orchestra.run.smithi129.stderr:INFO 2024-09-21 21:31:45,886 seastar - Reactor backend: linux-aio 2024-09-21T21:31:45.893 INFO:teuthology.orchestra.run.smithi129.stderr:==27681==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-21T21:31:46.000 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5fd5562 0x601b881 0x601ba89 0x601bc69 0x5f9bdb6 0x5f9bf90 0x5f9c51c 0x5de40ab 0x5dfc2ed 0x713a942 0x714c115 0x714d5b2 0x714d637 0x714d739 0x714d831 0x714da3e 0x714dc24 0x714df73 0x714e317 0x714e547 0x714e7b1 0x714ea7e 0x714ec9c 0x714ed54 0x714ee1a 0x714eefb 0x5c46eec 0x5c46ff6 0x5c470b2 0x5c4716e 0x5c4722a 0x5c47377 0x5c47502 0x5c478c3 0x5c616c6 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3961ae8 0x39632c2 0x3803c18 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:46.000 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:46.013 INFO:teuthology.orchestra.run.smithi129.stderr:INFO 2024-09-21 21:31:46,012 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-21T21:31:46.013 INFO:teuthology.orchestra.run.smithi129.stderr:INFO 2024-09-21 21:31:46,012 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 2-3 2024-09-21T21:31:46.014 INFO:teuthology.orchestra.run.smithi129.stderr:DEBUG 2024-09-21 21:31:46,013 [shard 0:main] seastar - reactor::drain 2024-09-21T21:31:46.515 INFO:teuthology.orchestra.run.smithi129.stderr:----------------------------------------------------- 2024-09-21T21:31:46.515 INFO:teuthology.orchestra.run.smithi129.stderr:Suppressions used: 2024-09-21T21:31:46.515 INFO:teuthology.orchestra.run.smithi129.stderr: count bytes template 2024-09-21T21:31:46.515 INFO:teuthology.orchestra.run.smithi129.stderr: 1 8 MallocExtension::Register 2024-09-21T21:31:46.515 INFO:teuthology.orchestra.run.smithi129.stderr:----------------------------------------------------- 2024-09-21T21:31:46.515 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-21T21:31:46.573 INFO:teuthology.orchestra.run.smithi129.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-21T21:31:46.577 INFO:teuthology.orchestra.run.smithi129.stderr:INFO 2024-09-21 21:31:46,576 seastar - Reactor backend: linux-aio 2024-09-21T21:31:46.590 INFO:teuthology.orchestra.run.smithi129.stderr:==27679==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-21T21:31:46.879 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xc622f 0x5c700ff 0xb9cd853 0xb9cdb54 0xb9cdcfc 0xb9cfab7 0xb9cfcb3 0xba14b9d 0xb9bc06f 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:46.879 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:46.963 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x3b8bdc1 0x3b8c15e 0xb9cc607 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9cfab7 0xb9cff02 0xb9bbe2c 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:47.021 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 123 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x124fe7 0x44735 0xfb4fb 0x3767e9b 0xb9aa945 0xb9aae1a 0xaca0a08 0x99e30fb 0x99e3957 0x98d9adc 0x9258389 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:47.021 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:47.052 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x10797f 0xdd2e1 0x4b14c 0x3b8bfdd 0x3b8c15e 0xb9cc607 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9cfab7 0xb9cfcb3 0xba14b9d 0xb9bc06f 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:47.052 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:47.167 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xdd323 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:47.168 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:47.225 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x9e16cc3 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:47.225 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:47.245 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4ab7c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:47.246 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:47.303 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4ab49 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:47.303 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:47.332 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x55250b7 0x9e16cf9 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:47.333 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:47.410 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 230 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525767 0x9e16d1b 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:47.411 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:48.930 INFO:teuthology.orchestra.run.smithi129.stderr:----------------------------------------------------- 2024-09-21T21:31:48.930 INFO:teuthology.orchestra.run.smithi129.stderr:Suppressions used: 2024-09-21T21:31:48.930 INFO:teuthology.orchestra.run.smithi129.stderr: count bytes template 2024-09-21T21:31:48.931 INFO:teuthology.orchestra.run.smithi129.stderr: 1 8 MallocExtension::Register 2024-09-21T21:31:48.931 INFO:teuthology.orchestra.run.smithi129.stderr:----------------------------------------------------- 2024-09-21T21:31:48.931 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-21T21:31:48.956 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-21T21:31:48.980 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T21:31:49.257 INFO:teuthology.orchestra.run.smithi129.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-21T21:31:49.258 INFO:teuthology.orchestra.run.smithi129.stderr:INFO 2024-09-21 21:31:49,257 seastar - Reactor backend: linux-aio 2024-09-21T21:31:49.263 INFO:teuthology.orchestra.run.smithi129.stderr:==27726==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-21T21:31:49.370 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4b137 0x5f94b4d 0x5f979bd 0x5f9c01a 0x5f9c51c 0x5de40ab 0x5dfc2ed 0x713a942 0x714c115 0x714d5b2 0x714d637 0x714d739 0x714d831 0x714da3e 0x714dc24 0x714df73 0x714e317 0x714e547 0x714e7b1 0x714ea7e 0x714ec9c 0x714ed54 0x714ee1a 0x714eefb 0x5c46eec 0x5c46ff6 0x5c470b2 0x5c4716e 0x5c4722a 0x5c47377 0x5c47502 0x5c478c3 0x5c616c6 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3961ae8 0x39632c2 0x3803c18 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:49.370 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:49.383 INFO:teuthology.orchestra.run.smithi129.stderr:INFO 2024-09-21 21:31:49,382 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-21T21:31:49.383 INFO:teuthology.orchestra.run.smithi129.stderr:INFO 2024-09-21 21:31:49,382 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 4-5 2024-09-21T21:31:49.384 INFO:teuthology.orchestra.run.smithi129.stderr:DEBUG 2024-09-21 21:31:49,383 [shard 0:main] seastar - reactor::drain 2024-09-21T21:31:49.888 INFO:teuthology.orchestra.run.smithi129.stderr:----------------------------------------------------- 2024-09-21T21:31:49.888 INFO:teuthology.orchestra.run.smithi129.stderr:Suppressions used: 2024-09-21T21:31:49.888 INFO:teuthology.orchestra.run.smithi129.stderr: count bytes template 2024-09-21T21:31:49.888 INFO:teuthology.orchestra.run.smithi129.stderr: 1 8 MallocExtension::Register 2024-09-21T21:31:49.888 INFO:teuthology.orchestra.run.smithi129.stderr:----------------------------------------------------- 2024-09-21T21:31:49.888 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-21T21:31:49.943 INFO:teuthology.orchestra.run.smithi129.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-21T21:31:49.947 INFO:teuthology.orchestra.run.smithi129.stderr:INFO 2024-09-21 21:31:49,946 seastar - Reactor backend: linux-aio 2024-09-21T21:31:49.961 INFO:teuthology.orchestra.run.smithi129.stderr:==27724==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-21T21:31:50.237 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x3868f3a 0xba09518 0xba0cf80 0xb9bfd1b 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:50.237 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:50.320 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4ab09 0xb9cc4a3 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9cfab7 0xb9cfcb3 0xba14b9d 0xb9bc06f 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:50.320 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:50.408 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xba1a76e 0xb9bba82 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:50.408 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:50.512 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x9e16d06 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:50.512 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:50.569 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4abc8 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:50.569 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:50.574 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xdd323 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:50.574 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:50.632 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525cde 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:50.632 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:50.677 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x9e16ce8 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:50.677 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:50.740 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 230 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x8b54f 0x10d76c 0xdd31a 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:50.740 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:51.227 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4aaff 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:51.228 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-09-21T21:31:52.221 INFO:teuthology.orchestra.run.smithi129.stderr:----------------------------------------------------- 2024-09-21T21:31:52.221 INFO:teuthology.orchestra.run.smithi129.stderr:Suppressions used: 2024-09-21T21:31:52.221 INFO:teuthology.orchestra.run.smithi129.stderr: count bytes template 2024-09-21T21:31:52.222 INFO:teuthology.orchestra.run.smithi129.stderr: 1 8 MallocExtension::Register 2024-09-21T21:31:52.222 INFO:teuthology.orchestra.run.smithi129.stderr:----------------------------------------------------- 2024-09-21T21:31:52.222 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-21T21:31:52.246 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T21:31:52.271 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-21T21:31:52.271 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T21:31:52.272 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-21T21:31:52.336 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T21:31:52.336 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-21T21:31:52.400 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T21:31:52.400 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-21T21:31:52.464 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T21:31:52.465 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-21T21:31:52.529 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T21:31:52.529 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-21T21:31:52.584 INFO:tasks.ceph:Adding keys to all mons... 2024-09-21T21:31:52.584 DEBUG:teuthology.orchestra.run.smithi129:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-21T21:31:52.647 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr.x] 2024-09-21T21:31:52.647 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQA7O+9m3ThQNBAAsj83mTLeIJkMD99fDaCxCA== 2024-09-21T21:31:52.647 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.0] 2024-09-21T21:31:52.647 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQA/O+9mw1bbFhAA7vdBA7UU4iPHkjZagpVi8g== 2024-09-21T21:31:52.648 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.1] 2024-09-21T21:31:52.648 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBCO+9m4UnhKxAA7IGuocm66LWj1ig7RMkl9A== 2024-09-21T21:31:52.648 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.2] 2024-09-21T21:31:52.648 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBGO+9mprQABhAAgWEe84Vsme8JmST8SZzu8A== 2024-09-21T21:31:52.648 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2024-09-21T21:31:52.648 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQA7O+9mpLx8NxAAGcMgNmXJIY6qnTCesObimw== 2024-09-21T21:31:52.649 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-21T21:31:52.751 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T21:31:52.812 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T21:31:52.873 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T21:31:52.934 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T21:31:52.998 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-21T21:31:52.998 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-21T21:31:53.022 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-21T21:31:53.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:31:53.150+0000 7f4384ed6c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:31:53.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:31:53.150+0000 7f4384ed6c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:31:53.165 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-21T21:31:53.190 DEBUG:teuthology.orchestra.run.smithi129:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-21T21:31:53.246 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-21T21:31:53.246 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-21T21:31:53.246 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-21T21:31:53.289 INFO:tasks.ceph.mon.a:Started 2024-09-21T21:31:53.289 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-21T21:31:53.289 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-21T21:31:53.289 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-21T21:31:53.291 INFO:tasks.ceph.mgr.x:Started 2024-09-21T21:31:53.291 DEBUG:tasks.ceph:set 0 configs 2024-09-21T21:31:53.291 INFO:tasks.ceph:Setting crush tunables to default 2024-09-21T21:31:53.292 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd crush tunables default 2024-09-21T21:31:53.371 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-21T21:31:53.369+0000 7f04b0a68c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:31:53.371 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-21T21:31:53.369+0000 7f04b0a68c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:31:53.371 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-21T21:31:53.369+0000 7f04b0a68c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:31:53.372 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-21T21:31:53.371+0000 7f20597bf000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:31:53.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:31:53.394+0000 7f5bb5191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:31:53.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:31:53.404+0000 7f5bb5191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:31:53.475 INFO:teuthology.orchestra.run.smithi129.stderr:adjusted tunables profile to default 2024-09-21T21:31:53.487 INFO:tasks.ceph:check_enable_crimson: True 2024-09-21T21:31:53.487 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-09-21T21:31:53.488 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-09-21T21:31:53.578 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-21T21:31:53.576+0000 7f20597bf000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:31:53.578 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-21T21:31:53.577+0000 7f20597bf000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:31:53.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:31:53.589+0000 7f35cf991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:31:53.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:31:53.597+0000 7f35cf991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:31:53.681 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-21T21:31:53.679+0000 7f04a89f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-09-21T21:31:53.681 INFO:tasks.ceph.mon.a.smithi129.stderr:Please be aware that this feature is experimental, untested, 2024-09-21T21:31:53.681 INFO:tasks.ceph.mon.a.smithi129.stderr:unsupported, and may result in data corruption, data loss, 2024-09-21T21:31:53.681 INFO:tasks.ceph.mon.a.smithi129.stderr:and/or irreparable damage to your cluster. Do not use 2024-09-21T21:31:53.681 INFO:tasks.ceph.mon.a.smithi129.stderr:feature with important data. 2024-09-21T21:31:53.681 INFO:tasks.ceph.mon.a.smithi129.stderr: 2024-09-21T21:31:54.488 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-21T21:31:54.523 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T21:31:54.524 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-21T21:31:54.549 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T21:31:54.549 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-21T21:31:54.615 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T21:31:54.615 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-21T21:31:54.679 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new 870c03e6-f121-4588-b0dc-6ce84b881ef5 0 2024-09-21T21:31:54.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:31:54.819+0000 7fa847391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:31:54.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:31:54.829+0000 7fa847391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:31:54.909 INFO:teuthology.orchestra.run.smithi129.stdout:0 2024-09-21T21:31:54.921 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new d2055726-f292-4619-94c7-cdd970552365 1 2024-09-21T21:31:55.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:31:55.021+0000 7fdc5887d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:31:55.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:31:55.031+0000 7fdc5887d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:31:55.111 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-09-21T21:31:55.122 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new 94e3907d-8e87-4ba8-bdd1-ad2eb5eecf12 2 2024-09-21T21:31:55.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:31:55.221+0000 7f204a2c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:31:55.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:31:55.231+0000 7f204a2c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:31:55.311 INFO:teuthology.orchestra.run.smithi129.stdout:2 2024-09-21T21:31:55.323 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-09-21T21:31:55.354 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-21T21:31:55.355 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-09-21T21:31:55.397 INFO:tasks.ceph.osd.0:Started 2024-09-21T21:31:55.397 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-09-21T21:31:55.427 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-21T21:31:55.428 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-21T21:31:55.470 INFO:tasks.ceph.osd.1:Started 2024-09-21T21:31:55.470 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-09-21T21:31:55.501 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-21T21:31:55.501 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-09-21T21:31:55.544 INFO:tasks.ceph.osd.2:Started 2024-09-21T21:31:55.544 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T21:31:55.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:31:55.705+0000 7fcec9d51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:31:55.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:31:55.715+0000 7fcec9d51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:31:55.788 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:31:55.788 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":6,"fsid":"9c86c2d1-86e4-4d70-bab7-28b6d262ee29","created":"2024-09-21T21:31:53.391647+0000","modified":"2024-09-21T21:31:55.309162+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T21:31:55.309162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"870c03e6-f121-4588-b0dc-6ce84b881ef5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d2055726-f292-4619-94c7-cdd970552365","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"94e3907d-8e87-4ba8-bdd1-ad2eb5eecf12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T21:31:55.800 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-21T21:31:55.800 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-21T21:31:56.101 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-21T21:31:56.207 INFO:teuthology.misc.health.smithi129.stderr:2024-09-21T21:31:56.201+0000 7f7b334ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:31:56.212 INFO:teuthology.misc.health.smithi129.stderr:2024-09-21T21:31:56.210+0000 7f7b334ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:31:56.280 INFO:teuthology.misc.health.smithi129.stdout: 2024-09-21T21:31:56.280 INFO:teuthology.misc.health.smithi129.stdout:{"epoch":6,"fsid":"9c86c2d1-86e4-4d70-bab7-28b6d262ee29","created":"2024-09-21T21:31:53.391647+0000","modified":"2024-09-21T21:31:55.309162+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T21:31:55.309162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"870c03e6-f121-4588-b0dc-6ce84b881ef5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d2055726-f292-4619-94c7-cdd970552365","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"94e3907d-8e87-4ba8-bdd1-ad2eb5eecf12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T21:31:56.291 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-21T21:31:59.034 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-21T21:31:59.032+0000 7f1ff695c640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-21T21:32:02.594 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-21T21:32:02.699 INFO:teuthology.misc.health.smithi129.stderr:2024-09-21T21:32:02.693+0000 7ffabc391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:02.703 INFO:teuthology.misc.health.smithi129.stderr:2024-09-21T21:32:02.702+0000 7ffabc391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:02.903 INFO:teuthology.misc.health.smithi129.stdout: 2024-09-21T21:32:02.904 INFO:teuthology.misc.health.smithi129.stdout:{"epoch":10,"fsid":"9c86c2d1-86e4-4d70-bab7-28b6d262ee29","created":"2024-09-21T21:31:53.391647+0000","modified":"2024-09-21T21:32:02.098232+0000","last_up_change":"2024-09-21T21:32:01.088108+0000","last_in_change":"2024-09-21T21:31:55.309162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T21:32:02.047960+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"870c03e6-f121-4588-b0dc-6ce84b881ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3419716534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6805","nonce":3419716534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":3419716534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":3419716534}]},"public_addr":"172.21.15.129:6804/3419716534","cluster_addr":"172.21.15.129:6805/3419716534","heartbeat_back_addr":"172.21.15.129:6812/3419716534","heartbeat_front_addr":"172.21.15.129:6813/3419716534","state":["exists","up"]},{"osd":1,"uuid":"d2055726-f292-4619-94c7-cdd970552365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6803","nonce":1486109390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":1486109390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6807","nonce":1486109390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":1486109390}]},"public_addr":"172.21.15.129:6803/1486109390","cluster_addr":"172.21.15.129:6802/1486109390","heartbeat_back_addr":"172.21.15.129:6807/1486109390","heartbeat_front_addr":"172.21.15.129:6806/1486109390","state":["exists","up"]},{"osd":2,"uuid":"94e3907d-8e87-4ba8-bdd1-ad2eb5eecf12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":3156771813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":3156771813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":3156771813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6811","nonce":3156771813}]},"public_addr":"172.21.15.129:6809/3156771813","cluster_addr":"172.21.15.129:6808/3156771813","heartbeat_back_addr":"172.21.15.129:6810/3156771813","heartbeat_front_addr":"172.21.15.129:6811/3156771813","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T21:32:02.917 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-21T21:32:02.917 INFO:tasks.ceph:Creating RBD pool 2024-09-21T21:32:02.917 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-21T21:32:03.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:03.022+0000 7f86c36bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:03.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:03.034+0000 7f86c36bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:04.108 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'rbd' created 2024-09-21T21:32:04.126 DEBUG:teuthology.orchestra.run.smithi129:> rbd --cluster ceph pool init rbd 2024-09-21T21:32:04.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:04.226+0000 7ff5a1c3eac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:04.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:04.226+0000 7ff5a1c3eac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:04.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:04.233+0000 7ff5a1c3eac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:07.240 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-21T21:32:07.240 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-21T21:32:07.241 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-21T21:32:07.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:07.344+0000 7fa8644d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:07.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:07.352+0000 7fa8644d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:07.542 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:32:07.555 INFO:teuthology.orchestra.run.smithi129.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2024-09-21T21:32:07.556 INFO:tasks.ceph_manager:config epoch is 1 2024-09-21T21:32:07.556 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T21:32:07.556 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-21T21:32:07.556 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-21T21:32:07.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:07.657+0000 7f35204a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:07.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:07.664+0000 7f35204a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:07.903 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:32:07.915 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2291442133},{"type":"v1","addr":"172.21.15.129:6801","nonce":2291442133}]},"active_addr":"172.21.15.129:6801/2291442133","active_change":"2024-09-21T21:31:58.017709+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.129:0","nonce":429052464}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.129:0","nonce":1449972350}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.129:0","nonce":1578979984}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.129:0","nonce":2816190528}]}]} 2024-09-21T21:32:07.917 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-21T21:32:07.917 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-21T21:32:07.918 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T21:32:08.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:08.019+0000 7f05e9791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:08.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:08.027+0000 7f05e9791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:08.218 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:32:08.219 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":15,"fsid":"9c86c2d1-86e4-4d70-bab7-28b6d262ee29","created":"2024-09-21T21:31:53.391647+0000","modified":"2024-09-21T21:32:07.167683+0000","last_up_change":"2024-09-21T21:32:01.088108+0000","last_in_change":"2024-09-21T21:31:55.309162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T21:32:02.047960+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T21:32:03.242947+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"870c03e6-f121-4588-b0dc-6ce84b881ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3419716534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6805","nonce":3419716534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":3419716534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":3419716534}]},"public_addr":"172.21.15.129:6804/3419716534","cluster_addr":"172.21.15.129:6805/3419716534","heartbeat_back_addr":"172.21.15.129:6812/3419716534","heartbeat_front_addr":"172.21.15.129:6813/3419716534","state":["exists","up"]},{"osd":1,"uuid":"d2055726-f292-4619-94c7-cdd970552365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6803","nonce":1486109390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":1486109390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6807","nonce":1486109390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":1486109390}]},"public_addr":"172.21.15.129:6803/1486109390","cluster_addr":"172.21.15.129:6802/1486109390","heartbeat_back_addr":"172.21.15.129:6807/1486109390","heartbeat_front_addr":"172.21.15.129:6806/1486109390","state":["exists","up"]},{"osd":2,"uuid":"94e3907d-8e87-4ba8-bdd1-ad2eb5eecf12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":3156771813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":3156771813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":3156771813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6811","nonce":3156771813}]},"public_addr":"172.21.15.129:6809/3156771813","cluster_addr":"172.21.15.129:6808/3156771813","heartbeat_back_addr":"172.21.15.129:6810/3156771813","heartbeat_front_addr":"172.21.15.129:6811/3156771813","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T21:32:08.231 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-21T21:32:08.232 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T21:32:08.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:08.333+0000 7f90ce991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:08.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:08.340+0000 7f90ce991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:08.530 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:32:08.530 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":15,"fsid":"9c86c2d1-86e4-4d70-bab7-28b6d262ee29","created":"2024-09-21T21:31:53.391647+0000","modified":"2024-09-21T21:32:07.167683+0000","last_up_change":"2024-09-21T21:32:01.088108+0000","last_in_change":"2024-09-21T21:31:55.309162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T21:32:02.047960+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T21:32:03.242947+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"870c03e6-f121-4588-b0dc-6ce84b881ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3419716534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6805","nonce":3419716534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":3419716534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":3419716534}]},"public_addr":"172.21.15.129:6804/3419716534","cluster_addr":"172.21.15.129:6805/3419716534","heartbeat_back_addr":"172.21.15.129:6812/3419716534","heartbeat_front_addr":"172.21.15.129:6813/3419716534","state":["exists","up"]},{"osd":1,"uuid":"d2055726-f292-4619-94c7-cdd970552365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6803","nonce":1486109390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":1486109390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6807","nonce":1486109390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":1486109390}]},"public_addr":"172.21.15.129:6803/1486109390","cluster_addr":"172.21.15.129:6802/1486109390","heartbeat_back_addr":"172.21.15.129:6807/1486109390","heartbeat_front_addr":"172.21.15.129:6806/1486109390","state":["exists","up"]},{"osd":2,"uuid":"94e3907d-8e87-4ba8-bdd1-ad2eb5eecf12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":3156771813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":3156771813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":3156771813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6811","nonce":3156771813}]},"public_addr":"172.21.15.129:6809/3156771813","cluster_addr":"172.21.15.129:6808/3156771813","heartbeat_back_addr":"172.21.15.129:6810/3156771813","heartbeat_front_addr":"172.21.15.129:6811/3156771813","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T21:32:08.544 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-21T21:32:08.544 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-21T21:32:08.545 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-21T21:32:08.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:08.652+0000 7f2075ad1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:08.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:08.652+0000 7ff44fcbd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:08.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:08.652+0000 7f8e68111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:08.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:08.661+0000 7ff44fcbd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:08.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:08.661+0000 7f2075ad1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:08.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:08.662+0000 7f8e68111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:08.692 INFO:teuthology.orchestra.run.smithi129.stdout:34359738376 2024-09-21T21:32:08.693 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T21:32:08.695 INFO:teuthology.orchestra.run.smithi129.stdout:38654705671 2024-09-21T21:32:08.695 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T21:32:08.696 INFO:teuthology.orchestra.run.smithi129.stdout:38654705671 2024-09-21T21:32:08.696 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T21:32:08.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:08.798+0000 7f302c85a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:08.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:08.809+0000 7f302c85a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:08.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:08.844+0000 7f7649cbd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:08.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:08.844+0000 7f7443391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:08.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:08.853+0000 7f7443391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:08.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:08.853+0000 7f7649cbd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:09.015 INFO:teuthology.orchestra.run.smithi129.stdout:34359738373 2024-09-21T21:32:09.028 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738376 got 34359738373 for osd.1 2024-09-21T21:32:09.071 INFO:teuthology.orchestra.run.smithi129.stdout:38654705668 2024-09-21T21:32:09.076 INFO:teuthology.orchestra.run.smithi129.stdout:38654705668 2024-09-21T21:32:09.088 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705668 for osd.0 2024-09-21T21:32:09.092 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705668 for osd.2 2024-09-21T21:32:10.029 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T21:32:10.089 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T21:32:10.093 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T21:32:10.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:10.148+0000 7fcc85751640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:10.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:10.158+0000 7fcc85751640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:10.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:10.202+0000 7f309e591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:10.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:10.213+0000 7f309e591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:10.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:10.244+0000 7fa4e9cc8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:10.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:10.253+0000 7fa4e9cc8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:10.361 INFO:teuthology.orchestra.run.smithi129.stdout:34359738376 2024-09-21T21:32:10.374 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738376 got 34359738376 for osd.1 2024-09-21T21:32:10.374 DEBUG:teuthology.parallel:result is None 2024-09-21T21:32:10.417 INFO:teuthology.orchestra.run.smithi129.stdout:38654705671 2024-09-21T21:32:10.430 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705671 for osd.0 2024-09-21T21:32:10.431 DEBUG:teuthology.parallel:result is None 2024-09-21T21:32:10.455 INFO:teuthology.orchestra.run.smithi129.stdout:38654705671 2024-09-21T21:32:10.468 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705671 for osd.2 2024-09-21T21:32:10.468 DEBUG:teuthology.parallel:result is None 2024-09-21T21:32:10.468 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-21T21:32:10.468 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T21:32:10.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:10.611+0000 7ff45bf91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:10.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:10.618+0000 7ff45bf91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:10.805 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:32:10.806 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-21T21:32:10.818 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-09-21T21:32:10.026312+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3372,"kb_used_data":3372,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889812,"statfs":{"total":160658620416,"available":160655167488,"internally_reserved":0,"allocated":3452928,"data_stored":3452928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.859733"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T21:32:07.195655+0000","last_change":"2024-09-21T21:32:07.204573+0000","last_active":"2024-09-21T21:32:07.195655+0000","last_peered":"2024-09-21T21:32:07.195655+0000","last_clean":"2024-09-21T21:32:07.195655+0000","last_became_active":"2024-09-21T21:32:05.222859+0000","last_became_peered":"2024-09-21T21:32:05.222859+0000","last_unstale":"2024-09-21T21:32:07.195655+0000","last_undegraded":"2024-09-21T21:32:07.195655+0000","last_fullsized":"2024-09-21T21:32:07.195655+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T21:32:07.202569+0000","last_change":"2024-09-21T21:32:07.212812+0000","last_active":"2024-09-21T21:32:07.202569+0000","last_peered":"2024-09-21T21:32:07.202569+0000","last_clean":"2024-09-21T21:32:07.202569+0000","last_became_active":"2024-09-21T21:32:05.255580+0000","last_became_peered":"2024-09-21T21:32:05.255580+0000","last_unstale":"2024-09-21T21:32:07.202569+0000","last_undegraded":"2024-09-21T21:32:07.202569+0000","last_fullsized":"2024-09-21T21:32:07.202569+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T21:32:07.193448+0000","last_change":"2024-09-21T21:32:07.204008+0000","last_active":"2024-09-21T21:32:07.193448+0000","last_peered":"2024-09-21T21:32:07.193448+0000","last_clean":"2024-09-21T21:32:07.193448+0000","last_became_active":"2024-09-21T21:32:05.237491+0000","last_became_peered":"2024-09-21T21:32:05.237491+0000","last_unstale":"2024-09-21T21:32:07.193448+0000","last_undegraded":"2024-09-21T21:32:07.193448+0000","last_fullsized":"2024-09-21T21:32:07.193448+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T21:32:07.200914+0000","last_change":"2024-09-21T21:32:07.213553+0000","last_active":"2024-09-21T21:32:07.200914+0000","last_peered":"2024-09-21T21:32:07.200914+0000","last_clean":"2024-09-21T21:32:07.200914+0000","last_became_active":"2024-09-21T21:32:05.299045+0000","last_became_peered":"2024-09-21T21:32:05.299045+0000","last_unstale":"2024-09-21T21:32:07.200914+0000","last_undegraded":"2024-09-21T21:32:07.200914+0000","last_fullsized":"2024-09-21T21:32:07.200914+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T21:32:07.210460+0000","last_change":"2024-09-21T21:32:07.210460+0000","last_active":"2024-09-21T21:32:07.210460+0000","last_peered":"2024-09-21T21:32:07.210460+0000","last_clean":"2024-09-21T21:32:07.210460+0000","last_became_active":"2024-09-21T21:32:05.275105+0000","last_became_peered":"2024-09-21T21:32:05.275105+0000","last_unstale":"2024-09-21T21:32:07.210460+0000","last_undegraded":"2024-09-21T21:32:07.210460+0000","last_fullsized":"2024-09-21T21:32:07.210460+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T21:32:07.207337+0000","last_change":"2024-09-21T21:32:07.212189+0000","last_active":"2024-09-21T21:32:07.207337+0000","last_peered":"2024-09-21T21:32:07.207337+0000","last_clean":"2024-09-21T21:32:07.207337+0000","last_became_active":"2024-09-21T21:32:05.304698+0000","last_became_peered":"2024-09-21T21:32:05.304698+0000","last_unstale":"2024-09-21T21:32:07.207337+0000","last_undegraded":"2024-09-21T21:32:07.207337+0000","last_fullsized":"2024-09-21T21:32:07.207337+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T21:32:07.200923+0000","last_change":"2024-09-21T21:32:07.205469+0000","last_active":"2024-09-21T21:32:07.200923+0000","last_peered":"2024-09-21T21:32:07.200923+0000","last_clean":"2024-09-21T21:32:07.200923+0000","last_became_active":"2024-09-21T21:32:05.258940+0000","last_became_peered":"2024-09-21T21:32:05.258940+0000","last_unstale":"2024-09-21T21:32:07.200923+0000","last_undegraded":"2024-09-21T21:32:07.200923+0000","last_fullsized":"2024-09-21T21:32:07.200923+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T21:32:07.206212+0000","last_change":"2024-09-21T21:32:07.213683+0000","last_active":"2024-09-21T21:32:07.206212+0000","last_peered":"2024-09-21T21:32:07.206212+0000","last_clean":"2024-09-21T21:32:07.206212+0000","last_became_active":"2024-09-21T21:32:05.219710+0000","last_became_peered":"2024-09-21T21:32:05.219710+0000","last_unstale":"2024-09-21T21:32:07.206212+0000","last_undegraded":"2024-09-21T21:32:07.206212+0000","last_fullsized":"2024-09-21T21:32:07.206212+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":87,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T21:32:07.196766+0000","last_change":"2024-09-21T21:32:03.188301+0000","last_active":"2024-09-21T21:32:07.196766+0000","last_peered":"2024-09-21T21:32:07.196766+0000","last_clean":"2024-09-21T21:32:07.196766+0000","last_became_active":"2024-09-21T21:32:03.168199+0000","last_became_peered":"2024-09-21T21:32:03.168199+0000","last_unstale":"2024-09-21T21:32:07.196766+0000","last_undegraded":"2024-09-21T21:32:07.196766+0000","last_fullsized":"2024-09-21T21:32:07.196766+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:02.098232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:02.098232+0000","last_clean_scrub_stamp":"2024-09-21T21:32:02.098232+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655167488,"internally_reserved":0,"allocated":3452928,"data_stored":3452928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":436,"kb_used_data":436,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297292,"statfs":{"total":53552873472,"available":53552427008,"internally_reserved":0,"allocated":446464,"data_stored":446464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738376,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1596,"kb_used_data":1596,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296132,"statfs":{"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1340,"kb_used_data":1340,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296388,"statfs":{"total":53552873472,"available":53551501312,"internally_reserved":0,"allocated":1372160,"data_stored":1372160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551501312,"internally_reserved":0,"allocated":1372160,"data_stored":1372160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552427008,"internally_reserved":0,"allocated":446464,"data_stored":446464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T21:32:10.819 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T21:32:10.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:10.921+0000 7f8179a46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:10.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:10.929+0000 7f8179a46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:11.114 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:32:11.114 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-21T21:32:11.127 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-09-21T21:32:10.026312+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3372,"kb_used_data":3372,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889812,"statfs":{"total":160658620416,"available":160655167488,"internally_reserved":0,"allocated":3452928,"data_stored":3452928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.859733"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T21:32:07.195655+0000","last_change":"2024-09-21T21:32:07.204573+0000","last_active":"2024-09-21T21:32:07.195655+0000","last_peered":"2024-09-21T21:32:07.195655+0000","last_clean":"2024-09-21T21:32:07.195655+0000","last_became_active":"2024-09-21T21:32:05.222859+0000","last_became_peered":"2024-09-21T21:32:05.222859+0000","last_unstale":"2024-09-21T21:32:07.195655+0000","last_undegraded":"2024-09-21T21:32:07.195655+0000","last_fullsized":"2024-09-21T21:32:07.195655+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T21:32:07.202569+0000","last_change":"2024-09-21T21:32:07.212812+0000","last_active":"2024-09-21T21:32:07.202569+0000","last_peered":"2024-09-21T21:32:07.202569+0000","last_clean":"2024-09-21T21:32:07.202569+0000","last_became_active":"2024-09-21T21:32:05.255580+0000","last_became_peered":"2024-09-21T21:32:05.255580+0000","last_unstale":"2024-09-21T21:32:07.202569+0000","last_undegraded":"2024-09-21T21:32:07.202569+0000","last_fullsized":"2024-09-21T21:32:07.202569+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T21:32:07.193448+0000","last_change":"2024-09-21T21:32:07.204008+0000","last_active":"2024-09-21T21:32:07.193448+0000","last_peered":"2024-09-21T21:32:07.193448+0000","last_clean":"2024-09-21T21:32:07.193448+0000","last_became_active":"2024-09-21T21:32:05.237491+0000","last_became_peered":"2024-09-21T21:32:05.237491+0000","last_unstale":"2024-09-21T21:32:07.193448+0000","last_undegraded":"2024-09-21T21:32:07.193448+0000","last_fullsized":"2024-09-21T21:32:07.193448+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T21:32:07.200914+0000","last_change":"2024-09-21T21:32:07.213553+0000","last_active":"2024-09-21T21:32:07.200914+0000","last_peered":"2024-09-21T21:32:07.200914+0000","last_clean":"2024-09-21T21:32:07.200914+0000","last_became_active":"2024-09-21T21:32:05.299045+0000","last_became_peered":"2024-09-21T21:32:05.299045+0000","last_unstale":"2024-09-21T21:32:07.200914+0000","last_undegraded":"2024-09-21T21:32:07.200914+0000","last_fullsized":"2024-09-21T21:32:07.200914+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T21:32:07.210460+0000","last_change":"2024-09-21T21:32:07.210460+0000","last_active":"2024-09-21T21:32:07.210460+0000","last_peered":"2024-09-21T21:32:07.210460+0000","last_clean":"2024-09-21T21:32:07.210460+0000","last_became_active":"2024-09-21T21:32:05.275105+0000","last_became_peered":"2024-09-21T21:32:05.275105+0000","last_unstale":"2024-09-21T21:32:07.210460+0000","last_undegraded":"2024-09-21T21:32:07.210460+0000","last_fullsized":"2024-09-21T21:32:07.210460+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T21:32:07.207337+0000","last_change":"2024-09-21T21:32:07.212189+0000","last_active":"2024-09-21T21:32:07.207337+0000","last_peered":"2024-09-21T21:32:07.207337+0000","last_clean":"2024-09-21T21:32:07.207337+0000","last_became_active":"2024-09-21T21:32:05.304698+0000","last_became_peered":"2024-09-21T21:32:05.304698+0000","last_unstale":"2024-09-21T21:32:07.207337+0000","last_undegraded":"2024-09-21T21:32:07.207337+0000","last_fullsized":"2024-09-21T21:32:07.207337+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T21:32:07.200923+0000","last_change":"2024-09-21T21:32:07.205469+0000","last_active":"2024-09-21T21:32:07.200923+0000","last_peered":"2024-09-21T21:32:07.200923+0000","last_clean":"2024-09-21T21:32:07.200923+0000","last_became_active":"2024-09-21T21:32:05.258940+0000","last_became_peered":"2024-09-21T21:32:05.258940+0000","last_unstale":"2024-09-21T21:32:07.200923+0000","last_undegraded":"2024-09-21T21:32:07.200923+0000","last_fullsized":"2024-09-21T21:32:07.200923+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T21:32:07.206212+0000","last_change":"2024-09-21T21:32:07.213683+0000","last_active":"2024-09-21T21:32:07.206212+0000","last_peered":"2024-09-21T21:32:07.206212+0000","last_clean":"2024-09-21T21:32:07.206212+0000","last_became_active":"2024-09-21T21:32:05.219710+0000","last_became_peered":"2024-09-21T21:32:05.219710+0000","last_unstale":"2024-09-21T21:32:07.206212+0000","last_undegraded":"2024-09-21T21:32:07.206212+0000","last_fullsized":"2024-09-21T21:32:07.206212+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":87,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T21:32:07.196766+0000","last_change":"2024-09-21T21:32:03.188301+0000","last_active":"2024-09-21T21:32:07.196766+0000","last_peered":"2024-09-21T21:32:07.196766+0000","last_clean":"2024-09-21T21:32:07.196766+0000","last_became_active":"2024-09-21T21:32:03.168199+0000","last_became_peered":"2024-09-21T21:32:03.168199+0000","last_unstale":"2024-09-21T21:32:07.196766+0000","last_undegraded":"2024-09-21T21:32:07.196766+0000","last_fullsized":"2024-09-21T21:32:07.196766+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:02.098232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:02.098232+0000","last_clean_scrub_stamp":"2024-09-21T21:32:02.098232+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655167488,"internally_reserved":0,"allocated":3452928,"data_stored":3452928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":436,"kb_used_data":436,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297292,"statfs":{"total":53552873472,"available":53552427008,"internally_reserved":0,"allocated":446464,"data_stored":446464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738376,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1596,"kb_used_data":1596,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296132,"statfs":{"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1340,"kb_used_data":1340,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296388,"statfs":{"total":53552873472,"available":53551501312,"internally_reserved":0,"allocated":1372160,"data_stored":1372160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551501312,"internally_reserved":0,"allocated":1372160,"data_stored":1372160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552427008,"internally_reserved":0,"allocated":446464,"data_stored":446464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T21:32:11.128 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-21T21:32:11.128 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T21:32:11.128 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-21T21:32:11.129 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-21T21:32:11.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:11.232+0000 7f24a2911640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:11.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:32:11.238+0000 7f24a2911640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:11.462 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:32:11.462 INFO:teuthology.orchestra.run.smithi129.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T21:32:11.476 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-21T21:32:11.476 INFO:teuthology.run_tasks:Running task workunit... 2024-09-21T21:32:11.488 INFO:tasks.workunit:Pulling workunits from ref 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T21:32:11.489 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-21T21:32:11.489 DEBUG:teuthology.orchestra.run.smithi129:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-21T21:32:11.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T21:32:11.504 INFO:teuthology.orchestra.run.smithi129.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-21T21:32:11.504 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-21T21:32:11.559 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-21T21:32:11.559 DEBUG:teuthology.orchestra.run.smithi129:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-21T21:32:11.614 INFO:tasks.workunit:timeout=3h 2024-09-21T21:32:11.614 INFO:tasks.workunit:cleanup=True 2024-09-21T21:32:11.615 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T21:32:11.670 INFO:tasks.workunit.client.0.smithi129.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-21T21:33:07.285 INFO:tasks.workunit.client.0.smithi129.stderr:Note: switching to '62da54b4b83a96ca9976b7e1d0c8272a564a2208'. 2024-09-21T21:33:07.285 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-09-21T21:33:07.286 INFO:tasks.workunit.client.0.smithi129.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-21T21:33:07.286 INFO:tasks.workunit.client.0.smithi129.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-21T21:33:07.286 INFO:tasks.workunit.client.0.smithi129.stderr:state without impacting any branches by switching back to a branch. 2024-09-21T21:33:07.286 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-09-21T21:33:07.286 INFO:tasks.workunit.client.0.smithi129.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-21T21:33:07.286 INFO:tasks.workunit.client.0.smithi129.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-21T21:33:07.286 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-09-21T21:33:07.286 INFO:tasks.workunit.client.0.smithi129.stderr: git switch -c 2024-09-21T21:33:07.286 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-09-21T21:33:07.286 INFO:tasks.workunit.client.0.smithi129.stderr:Or undo this operation with: 2024-09-21T21:33:07.286 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-09-21T21:33:07.286 INFO:tasks.workunit.client.0.smithi129.stderr: git switch - 2024-09-21T21:33:07.287 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-09-21T21:33:07.287 INFO:tasks.workunit.client.0.smithi129.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-21T21:33:07.287 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-09-21T21:33:07.287 INFO:tasks.workunit.client.0.smithi129.stderr:HEAD is now at 62da54b4b83 Merge pull request #59904 from mcv21/patch-2 2024-09-21T21:33:07.288 DEBUG:teuthology.orchestra.run.smithi129:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-21T21:33:07.349 INFO:tasks.workunit.client.0.smithi129.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-21T21:33:07.351 INFO:tasks.workunit.client.0.smithi129.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T21:33:07.351 INFO:tasks.workunit.client.0.smithi129.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-21T21:33:08.006 INFO:tasks.workunit.client.0.smithi129.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-21T21:33:08.068 INFO:tasks.workunit.client.0.smithi129.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-21T21:33:08.109 INFO:tasks.workunit.client.0.smithi129.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T21:33:08.110 INFO:tasks.workunit.client.0.smithi129.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T21:33:08.111 INFO:tasks.workunit.client.0.smithi129.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-21T21:33:08.149 INFO:tasks.workunit.client.0.smithi129.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T21:33:08.152 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T21:33:08.153 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-21T21:33:08.209 INFO:tasks.workunit:Running workunits matching rbd/test_lock_fence.sh on client.0... 2024-09-21T21:33:08.210 INFO:tasks.workunit:Running workunit rbd/test_lock_fence.sh... 2024-09-21T21:33:08.210 DEBUG:teuthology.orchestra.run.smithi129:workunit test rbd/test_lock_fence.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=62da54b4b83a96ca9976b7e1d0c8272a564a2208 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_lock_fence.sh 2024-09-21T21:33:08.269 INFO:tasks.workunit.client.0.smithi129.stderr:+ IMAGE=rbdrw-image 2024-09-21T21:33:08.269 INFO:tasks.workunit.client.0.smithi129.stderr:+ LOCKID=rbdrw 2024-09-21T21:33:08.269 INFO:tasks.workunit.client.0.smithi129.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_lock_fence.sh 2024-09-21T21:33:08.270 INFO:tasks.workunit.client.0.smithi129.stderr:+ RELPATH=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/librbd 2024-09-21T21:33:08.270 INFO:tasks.workunit.client.0.smithi129.stderr:+ RBDRW=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/librbd/rbdrw.py 2024-09-21T21:33:08.270 INFO:tasks.workunit.client.0.smithi129.stderr:+ rbd create rbdrw-image --size 10 --image-format 2 --image-shared 2024-09-21T21:33:08.312 INFO:tasks.workunit.client.0.smithi129.stderr:2024-09-21T21:33:08.310+0000 7f751654dac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:08.313 INFO:tasks.workunit.client.0.smithi129.stderr:2024-09-21T21:33:08.310+0000 7f751654dac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:08.317 INFO:tasks.workunit.client.0.smithi129.stderr:2024-09-21T21:33:08.315+0000 7f751654dac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:08.428 INFO:tasks.workunit.client.0.smithi129.stderr:+ iochild=29744 2024-09-21T21:33:08.428 INFO:tasks.workunit.client.0.smithi129.stderr:+ LOCKS='[]' 2024-09-21T21:33:08.428 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' '[]' == '[]' ']' 2024-09-21T21:33:08.428 INFO:tasks.workunit.client.0.smithi129.stderr:+ python3 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/librbd/rbdrw.py rbdrw-image rbdrw 2024-09-21T21:33:08.428 INFO:tasks.workunit.client.0.smithi129.stderr:++ rbd lock list rbdrw-image --format json 2024-09-21T21:33:08.470 INFO:tasks.workunit.client.0.smithi129.stderr:2024-09-21T21:33:08.467+0000 7f4b31e48ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:08.470 INFO:tasks.workunit.client.0.smithi129.stderr:2024-09-21T21:33:08.467+0000 7f4b31e48ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:08.475 INFO:tasks.workunit.client.0.smithi129.stderr:2024-09-21T21:33:08.472+0000 7f4b31e48ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:08.626 INFO:tasks.workunit.client.0.smithi129.stderr:+ LOCKS='[]' 2024-09-21T21:33:08.626 INFO:tasks.workunit.client.0.smithi129.stderr:+ sleep 1 2024-09-21T21:33:09.627 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' '[]' == '[]' ']' 2024-09-21T21:33:09.627 INFO:tasks.workunit.client.0.smithi129.stderr:++ rbd lock list rbdrw-image --format json 2024-09-21T21:33:09.671 INFO:tasks.workunit.client.0.smithi129.stderr:2024-09-21T21:33:09.669+0000 7f2094810ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:09.671 INFO:tasks.workunit.client.0.smithi129.stderr:2024-09-21T21:33:09.669+0000 7f2094810ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:09.676 INFO:tasks.workunit.client.0.smithi129.stderr:2024-09-21T21:33:09.674+0000 7f2094810ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:09.826 INFO:tasks.workunit.client.0.smithi129.stderr:+ LOCKS='[{"id":"rbdrw","locker":"client.4204","address":"172.21.15.129:0/2282313657"}]' 2024-09-21T21:33:09.826 INFO:tasks.workunit.client.0.smithi129.stderr:+ sleep 1 2024-09-21T21:33:10.827 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' '[{"id":"rbdrw","locker":"client.4204","address":"172.21.15.129:0/2282313657"}]' == '[]' ']' 2024-09-21T21:33:10.827 INFO:tasks.workunit.client.0.smithi129.stderr:++ rbd lock list rbdrw-image 2024-09-21T21:33:10.827 INFO:tasks.workunit.client.0.smithi129.stderr:++ tail -1 2024-09-21T21:33:10.828 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $NF;}' 2024-09-21T21:33:10.868 INFO:tasks.workunit.client.0.smithi129.stderr:2024-09-21T21:33:10.865+0000 7fb4a5840ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:10.868 INFO:tasks.workunit.client.0.smithi129.stderr:2024-09-21T21:33:10.865+0000 7fb4a5840ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:10.873 INFO:tasks.workunit.client.0.smithi129.stderr:2024-09-21T21:33:10.871+0000 7fb4a5840ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:11.051 INFO:tasks.workunit.client.0.smithi129.stderr:+ clientaddr=172.21.15.129:0/2282313657 2024-09-21T21:33:11.051 INFO:tasks.workunit.client.0.smithi129.stderr:++ rbd lock list rbdrw-image 2024-09-21T21:33:11.052 INFO:tasks.workunit.client.0.smithi129.stderr:++ tail -1 2024-09-21T21:33:11.052 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1;}' 2024-09-21T21:33:11.092 INFO:tasks.workunit.client.0.smithi129.stderr:2024-09-21T21:33:11.090+0000 7f075cdb9ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:11.093 INFO:tasks.workunit.client.0.smithi129.stderr:2024-09-21T21:33:11.090+0000 7f075cdb9ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:11.097 INFO:tasks.workunit.client.0.smithi129.stderr:2024-09-21T21:33:11.095+0000 7f075cdb9ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:11.251 INFO:tasks.workunit.client.0.smithi129.stderr:+ clientid=client.4204 2024-09-21T21:33:11.252 INFO:tasks.workunit.client.0.smithi129.stdout:clientaddr: 172.21.15.129:0/2282313657 2024-09-21T21:33:11.252 INFO:tasks.workunit.client.0.smithi129.stdout:clientid: client.4204 2024-09-21T21:33:11.252 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'clientaddr: 172.21.15.129:0/2282313657' 2024-09-21T21:33:11.252 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'clientid: client.4204' 2024-09-21T21:33:11.252 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph osd blocklist add 172.21.15.129:0/2282313657 2024-09-21T21:33:11.353 INFO:tasks.workunit.client.0.smithi129.stderr:2024-09-21T21:33:11.343+0000 7f0453191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:11.357 INFO:tasks.workunit.client.0.smithi129.stderr:2024-09-21T21:33:11.354+0000 7f0453191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:13.416 INFO:tasks.workunit.client.0.smithi129.stderr:blocklisting 172.21.15.129:0/2282313657 until 2024-09-21T22:33:12.596353+0000 (3600 sec) 2024-09-21T21:33:13.439 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 29744 2024-09-21T21:33:13.439 INFO:tasks.workunit.client.0.smithi129.stderr:+ rbdrw_exitcode=108 2024-09-21T21:33:13.439 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 108 '!=' 108 ']' 2024-09-21T21:33:13.439 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'rbdrw stopped with ESHUTDOWN' 2024-09-21T21:33:13.440 INFO:tasks.workunit.client.0.smithi129.stdout:rbdrw stopped with ESHUTDOWN 2024-09-21T21:33:13.440 INFO:tasks.workunit.client.0.smithi129.stderr:+ set -e 2024-09-21T21:33:13.440 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph osd blocklist rm 172.21.15.129:0/2282313657 2024-09-21T21:33:13.538 INFO:tasks.workunit.client.0.smithi129.stderr:2024-09-21T21:33:13.528+0000 7fecdb191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:13.539 INFO:tasks.workunit.client.0.smithi129.stderr:2024-09-21T21:33:13.537+0000 7fecdb191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:14.546 INFO:tasks.workunit.client.0.smithi129.stderr:172.21.15.129:0/2282313657 isn't blocklisted 2024-09-21T21:33:14.558 INFO:tasks.workunit.client.0.smithi129.stderr:+ rbd lock remove rbdrw-image rbdrw client.4204 2024-09-21T21:33:14.599 INFO:tasks.workunit.client.0.smithi129.stderr:2024-09-21T21:33:14.597+0000 7f2402a51ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:14.599 INFO:tasks.workunit.client.0.smithi129.stderr:2024-09-21T21:33:14.597+0000 7f2402a51ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:14.603 INFO:tasks.workunit.client.0.smithi129.stderr:2024-09-21T21:33:14.602+0000 7f2402a51ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:15.502 INFO:tasks.workunit.client.0.smithi129.stderr:+ sleep 30 2024-09-21T21:33:45.504 INFO:tasks.workunit.client.0.smithi129.stderr:+ rbd rm rbdrw-image 2024-09-21T21:33:45.546 INFO:tasks.workunit.client.0.smithi129.stderr:2024-09-21T21:33:45.544+0000 7ff1e745dac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:45.546 INFO:tasks.workunit.client.0.smithi129.stderr:2024-09-21T21:33:45.544+0000 7ff1e745dac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:45.551 INFO:tasks.workunit.client.0.smithi129.stderr:2024-09-21T21:33:45.549+0000 7ff1e745dac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:46.205 INFO:tasks.workunit.client.0.smithi129.stderr: Removing image: 33% complete... Removing image: 66% complete... Removing image: 100% complete...done. 2024-09-21T21:33:46.209 INFO:tasks.workunit.client.0.smithi129.stdout:OK 2024-09-21T21:33:46.210 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo OK 2024-09-21T21:33:46.211 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-21T21:33:46.211 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-21T21:33:46.275 INFO:tasks.workunit:Stopping ['rbd/test_lock_fence.sh'] on client.0... 2024-09-21T21:33:46.276 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-21T21:33:46.613 DEBUG:teuthology.parallel:result is None 2024-09-21T21:33:46.614 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T21:33:46.639 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T21:33:46.639 DEBUG:teuthology.orchestra.run.smithi129:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-21T21:33:46.707 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T21:33:46.707 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-21T21:33:46.717 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-21T21:33:46.718 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T21:33:46.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:33:46.848+0000 7f8c46991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:46.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:33:46.855+0000 7f8c46991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:47.040 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:33:47.041 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-21T21:33:47.054 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-09-21T21:33:46.047072+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594499,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":377,"num_read_kb":595,"num_write":219,"num_write_kb":1643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":197,"ondisk_log_size":197,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":4276,"kb_used_data":4276,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156888908,"statfs":{"total":160658620416,"available":160654241792,"internally_reserved":0,"allocated":4378624,"data_stored":4378624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003191"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.447208+0000","last_change":"2024-09-21T21:32:07.204573+0000","last_active":"2024-09-21T21:33:15.447208+0000","last_peered":"2024-09-21T21:33:15.447208+0000","last_clean":"2024-09-21T21:33:15.447208+0000","last_became_active":"2024-09-21T21:32:05.222859+0000","last_became_peered":"2024-09-21T21:32:05.222859+0000","last_unstale":"2024-09-21T21:33:15.447208+0000","last_undegraded":"2024-09-21T21:33:15.447208+0000","last_fullsized":"2024-09-21T21:33:15.447208+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.452562+0000","last_change":"2024-09-21T21:32:07.212812+0000","last_active":"2024-09-21T21:33:15.452562+0000","last_peered":"2024-09-21T21:33:15.452562+0000","last_clean":"2024-09-21T21:33:15.452562+0000","last_became_active":"2024-09-21T21:32:05.255580+0000","last_became_peered":"2024-09-21T21:32:05.255580+0000","last_unstale":"2024-09-21T21:33:15.452562+0000","last_undegraded":"2024-09-21T21:33:15.452562+0000","last_fullsized":"2024-09-21T21:33:15.452562+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"19'8","reported_seq":104,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.480255+0000","last_change":"2024-09-21T21:32:07.204008+0000","last_active":"2024-09-21T21:33:15.480255+0000","last_peered":"2024-09-21T21:33:15.480255+0000","last_clean":"2024-09-21T21:33:15.480255+0000","last_became_active":"2024-09-21T21:32:05.237491+0000","last_became_peered":"2024-09-21T21:32:05.237491+0000","last_unstale":"2024-09-21T21:33:15.480255+0000","last_undegraded":"2024-09-21T21:33:15.480255+0000","last_fullsized":"2024-09-21T21:33:15.480255+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":181,"num_read_kb":20,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"15'1","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.453141+0000","last_change":"2024-09-21T21:32:07.213553+0000","last_active":"2024-09-21T21:33:15.453141+0000","last_peered":"2024-09-21T21:33:15.453141+0000","last_clean":"2024-09-21T21:33:15.453141+0000","last_became_active":"2024-09-21T21:32:05.299045+0000","last_became_peered":"2024-09-21T21:32:05.299045+0000","last_unstale":"2024-09-21T21:33:15.453141+0000","last_undegraded":"2024-09-21T21:33:15.453141+0000","last_fullsized":"2024-09-21T21:33:15.453141+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'135","reported_seq":303,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.453720+0000","last_change":"2024-09-21T21:32:07.210460+0000","last_active":"2024-09-21T21:33:15.453720+0000","last_peered":"2024-09-21T21:33:15.453720+0000","last_clean":"2024-09-21T21:33:15.453720+0000","last_became_active":"2024-09-21T21:32:05.275105+0000","last_became_peered":"2024-09-21T21:32:05.275105+0000","last_unstale":"2024-09-21T21:33:15.453720+0000","last_undegraded":"2024-09-21T21:33:15.453720+0000","last_fullsized":"2024-09-21T21:33:15.453720+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4115,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":141,"num_read_kb":529,"num_write":135,"num_write_kb":534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.452135+0000","last_change":"2024-09-21T21:32:07.212189+0000","last_active":"2024-09-21T21:33:15.452135+0000","last_peered":"2024-09-21T21:33:15.452135+0000","last_clean":"2024-09-21T21:33:15.452135+0000","last_became_active":"2024-09-21T21:32:05.304698+0000","last_became_peered":"2024-09-21T21:32:05.304698+0000","last_unstale":"2024-09-21T21:33:15.452135+0000","last_undegraded":"2024-09-21T21:33:15.452135+0000","last_fullsized":"2024-09-21T21:33:15.452135+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.450090+0000","last_change":"2024-09-21T21:32:07.205469+0000","last_active":"2024-09-21T21:33:15.450090+0000","last_peered":"2024-09-21T21:33:15.450090+0000","last_clean":"2024-09-21T21:33:15.450090+0000","last_became_active":"2024-09-21T21:32:05.258940+0000","last_became_peered":"2024-09-21T21:32:05.258940+0000","last_unstale":"2024-09-21T21:33:15.450090+0000","last_undegraded":"2024-09-21T21:33:15.450090+0000","last_fullsized":"2024-09-21T21:33:15.450090+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.451883+0000","last_change":"2024-09-21T21:32:07.213683+0000","last_active":"2024-09-21T21:33:15.451883+0000","last_peered":"2024-09-21T21:33:15.451883+0000","last_clean":"2024-09-21T21:33:15.451883+0000","last_became_active":"2024-09-21T21:32:05.219710+0000","last_became_peered":"2024-09-21T21:32:05.219710+0000","last_unstale":"2024-09-21T21:33:15.451883+0000","last_undegraded":"2024-09-21T21:33:15.451883+0000","last_fullsized":"2024-09-21T21:33:15.451883+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":99,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.446684+0000","last_change":"2024-09-21T21:32:03.188301+0000","last_active":"2024-09-21T21:33:15.446684+0000","last_peered":"2024-09-21T21:33:15.446684+0000","last_clean":"2024-09-21T21:33:15.446684+0000","last_became_active":"2024-09-21T21:32:03.168199+0000","last_became_peered":"2024-09-21T21:32:03.168199+0000","last_unstale":"2024-09-21T21:33:15.446684+0000","last_undegraded":"2024-09-21T21:33:15.446684+0000","last_fullsized":"2024-09-21T21:33:15.446684+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:02.098232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:02.098232+0000","last_clean_scrub_stamp":"2024-09-21T21:32:02.098232+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4131,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":324,"num_read_kb":549,"num_write":140,"num_write_kb":537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160654241792,"internally_reserved":0,"allocated":4378624,"data_stored":4378624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53550780416,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":468,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297260,"statfs":{"total":53552873472,"available":53552394240,"internally_reserved":0,"allocated":479232,"data_stored":479232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738468,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2044,"kb_used_data":2044,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295684,"statfs":{"total":53552873472,"available":53550780416,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705763,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1764,"kb_used_data":1764,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295964,"statfs":{"total":53552873472,"available":53551067136,"internally_reserved":0,"allocated":1806336,"data_stored":1806336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53550780416,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551067136,"internally_reserved":0,"allocated":1806336,"data_stored":1806336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550780416,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552394240,"internally_reserved":0,"allocated":479232,"data_stored":479232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T21:33:47.054 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T21:33:47.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:33:47.157+0000 7f0afe2ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:47.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:33:47.165+0000 7f0afe2ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:47.354 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:33:47.355 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-21T21:33:47.367 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-09-21T21:33:46.047072+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594499,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":377,"num_read_kb":595,"num_write":219,"num_write_kb":1643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":197,"ondisk_log_size":197,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":4276,"kb_used_data":4276,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156888908,"statfs":{"total":160658620416,"available":160654241792,"internally_reserved":0,"allocated":4378624,"data_stored":4378624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003191"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.447208+0000","last_change":"2024-09-21T21:32:07.204573+0000","last_active":"2024-09-21T21:33:15.447208+0000","last_peered":"2024-09-21T21:33:15.447208+0000","last_clean":"2024-09-21T21:33:15.447208+0000","last_became_active":"2024-09-21T21:32:05.222859+0000","last_became_peered":"2024-09-21T21:32:05.222859+0000","last_unstale":"2024-09-21T21:33:15.447208+0000","last_undegraded":"2024-09-21T21:33:15.447208+0000","last_fullsized":"2024-09-21T21:33:15.447208+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.452562+0000","last_change":"2024-09-21T21:32:07.212812+0000","last_active":"2024-09-21T21:33:15.452562+0000","last_peered":"2024-09-21T21:33:15.452562+0000","last_clean":"2024-09-21T21:33:15.452562+0000","last_became_active":"2024-09-21T21:32:05.255580+0000","last_became_peered":"2024-09-21T21:32:05.255580+0000","last_unstale":"2024-09-21T21:33:15.452562+0000","last_undegraded":"2024-09-21T21:33:15.452562+0000","last_fullsized":"2024-09-21T21:33:15.452562+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"19'8","reported_seq":104,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.480255+0000","last_change":"2024-09-21T21:32:07.204008+0000","last_active":"2024-09-21T21:33:15.480255+0000","last_peered":"2024-09-21T21:33:15.480255+0000","last_clean":"2024-09-21T21:33:15.480255+0000","last_became_active":"2024-09-21T21:32:05.237491+0000","last_became_peered":"2024-09-21T21:32:05.237491+0000","last_unstale":"2024-09-21T21:33:15.480255+0000","last_undegraded":"2024-09-21T21:33:15.480255+0000","last_fullsized":"2024-09-21T21:33:15.480255+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":181,"num_read_kb":20,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"15'1","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.453141+0000","last_change":"2024-09-21T21:32:07.213553+0000","last_active":"2024-09-21T21:33:15.453141+0000","last_peered":"2024-09-21T21:33:15.453141+0000","last_clean":"2024-09-21T21:33:15.453141+0000","last_became_active":"2024-09-21T21:32:05.299045+0000","last_became_peered":"2024-09-21T21:32:05.299045+0000","last_unstale":"2024-09-21T21:33:15.453141+0000","last_undegraded":"2024-09-21T21:33:15.453141+0000","last_fullsized":"2024-09-21T21:33:15.453141+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'135","reported_seq":303,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.453720+0000","last_change":"2024-09-21T21:32:07.210460+0000","last_active":"2024-09-21T21:33:15.453720+0000","last_peered":"2024-09-21T21:33:15.453720+0000","last_clean":"2024-09-21T21:33:15.453720+0000","last_became_active":"2024-09-21T21:32:05.275105+0000","last_became_peered":"2024-09-21T21:32:05.275105+0000","last_unstale":"2024-09-21T21:33:15.453720+0000","last_undegraded":"2024-09-21T21:33:15.453720+0000","last_fullsized":"2024-09-21T21:33:15.453720+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4115,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":141,"num_read_kb":529,"num_write":135,"num_write_kb":534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.452135+0000","last_change":"2024-09-21T21:32:07.212189+0000","last_active":"2024-09-21T21:33:15.452135+0000","last_peered":"2024-09-21T21:33:15.452135+0000","last_clean":"2024-09-21T21:33:15.452135+0000","last_became_active":"2024-09-21T21:32:05.304698+0000","last_became_peered":"2024-09-21T21:32:05.304698+0000","last_unstale":"2024-09-21T21:33:15.452135+0000","last_undegraded":"2024-09-21T21:33:15.452135+0000","last_fullsized":"2024-09-21T21:33:15.452135+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.450090+0000","last_change":"2024-09-21T21:32:07.205469+0000","last_active":"2024-09-21T21:33:15.450090+0000","last_peered":"2024-09-21T21:33:15.450090+0000","last_clean":"2024-09-21T21:33:15.450090+0000","last_became_active":"2024-09-21T21:32:05.258940+0000","last_became_peered":"2024-09-21T21:32:05.258940+0000","last_unstale":"2024-09-21T21:33:15.450090+0000","last_undegraded":"2024-09-21T21:33:15.450090+0000","last_fullsized":"2024-09-21T21:33:15.450090+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.451883+0000","last_change":"2024-09-21T21:32:07.213683+0000","last_active":"2024-09-21T21:33:15.451883+0000","last_peered":"2024-09-21T21:33:15.451883+0000","last_clean":"2024-09-21T21:33:15.451883+0000","last_became_active":"2024-09-21T21:32:05.219710+0000","last_became_peered":"2024-09-21T21:32:05.219710+0000","last_unstale":"2024-09-21T21:33:15.451883+0000","last_undegraded":"2024-09-21T21:33:15.451883+0000","last_fullsized":"2024-09-21T21:33:15.451883+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":99,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.446684+0000","last_change":"2024-09-21T21:32:03.188301+0000","last_active":"2024-09-21T21:33:15.446684+0000","last_peered":"2024-09-21T21:33:15.446684+0000","last_clean":"2024-09-21T21:33:15.446684+0000","last_became_active":"2024-09-21T21:32:03.168199+0000","last_became_peered":"2024-09-21T21:32:03.168199+0000","last_unstale":"2024-09-21T21:33:15.446684+0000","last_undegraded":"2024-09-21T21:33:15.446684+0000","last_fullsized":"2024-09-21T21:33:15.446684+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:02.098232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:02.098232+0000","last_clean_scrub_stamp":"2024-09-21T21:32:02.098232+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4131,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":324,"num_read_kb":549,"num_write":140,"num_write_kb":537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160654241792,"internally_reserved":0,"allocated":4378624,"data_stored":4378624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53550780416,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":468,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297260,"statfs":{"total":53552873472,"available":53552394240,"internally_reserved":0,"allocated":479232,"data_stored":479232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738468,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2044,"kb_used_data":2044,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295684,"statfs":{"total":53552873472,"available":53550780416,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705763,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1764,"kb_used_data":1764,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295964,"statfs":{"total":53552873472,"available":53551067136,"internally_reserved":0,"allocated":1806336,"data_stored":1806336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53550780416,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551067136,"internally_reserved":0,"allocated":1806336,"data_stored":1806336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550780416,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552394240,"internally_reserved":0,"allocated":479232,"data_stored":479232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T21:33:47.368 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-21T21:33:47.368 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T21:33:47.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:33:47.470+0000 7f6e68f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:47.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:33:47.477+0000 7f6e68f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:47.665 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:33:47.666 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-21T21:33:47.678 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-09-21T21:33:46.047072+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594499,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":377,"num_read_kb":595,"num_write":219,"num_write_kb":1643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":197,"ondisk_log_size":197,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":4276,"kb_used_data":4276,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156888908,"statfs":{"total":160658620416,"available":160654241792,"internally_reserved":0,"allocated":4378624,"data_stored":4378624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003191"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.447208+0000","last_change":"2024-09-21T21:32:07.204573+0000","last_active":"2024-09-21T21:33:15.447208+0000","last_peered":"2024-09-21T21:33:15.447208+0000","last_clean":"2024-09-21T21:33:15.447208+0000","last_became_active":"2024-09-21T21:32:05.222859+0000","last_became_peered":"2024-09-21T21:32:05.222859+0000","last_unstale":"2024-09-21T21:33:15.447208+0000","last_undegraded":"2024-09-21T21:33:15.447208+0000","last_fullsized":"2024-09-21T21:33:15.447208+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.452562+0000","last_change":"2024-09-21T21:32:07.212812+0000","last_active":"2024-09-21T21:33:15.452562+0000","last_peered":"2024-09-21T21:33:15.452562+0000","last_clean":"2024-09-21T21:33:15.452562+0000","last_became_active":"2024-09-21T21:32:05.255580+0000","last_became_peered":"2024-09-21T21:32:05.255580+0000","last_unstale":"2024-09-21T21:33:15.452562+0000","last_undegraded":"2024-09-21T21:33:15.452562+0000","last_fullsized":"2024-09-21T21:33:15.452562+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"19'8","reported_seq":104,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.480255+0000","last_change":"2024-09-21T21:32:07.204008+0000","last_active":"2024-09-21T21:33:15.480255+0000","last_peered":"2024-09-21T21:33:15.480255+0000","last_clean":"2024-09-21T21:33:15.480255+0000","last_became_active":"2024-09-21T21:32:05.237491+0000","last_became_peered":"2024-09-21T21:32:05.237491+0000","last_unstale":"2024-09-21T21:33:15.480255+0000","last_undegraded":"2024-09-21T21:33:15.480255+0000","last_fullsized":"2024-09-21T21:33:15.480255+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":181,"num_read_kb":20,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"15'1","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.453141+0000","last_change":"2024-09-21T21:32:07.213553+0000","last_active":"2024-09-21T21:33:15.453141+0000","last_peered":"2024-09-21T21:33:15.453141+0000","last_clean":"2024-09-21T21:33:15.453141+0000","last_became_active":"2024-09-21T21:32:05.299045+0000","last_became_peered":"2024-09-21T21:32:05.299045+0000","last_unstale":"2024-09-21T21:33:15.453141+0000","last_undegraded":"2024-09-21T21:33:15.453141+0000","last_fullsized":"2024-09-21T21:33:15.453141+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'135","reported_seq":303,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.453720+0000","last_change":"2024-09-21T21:32:07.210460+0000","last_active":"2024-09-21T21:33:15.453720+0000","last_peered":"2024-09-21T21:33:15.453720+0000","last_clean":"2024-09-21T21:33:15.453720+0000","last_became_active":"2024-09-21T21:32:05.275105+0000","last_became_peered":"2024-09-21T21:32:05.275105+0000","last_unstale":"2024-09-21T21:33:15.453720+0000","last_undegraded":"2024-09-21T21:33:15.453720+0000","last_fullsized":"2024-09-21T21:33:15.453720+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4115,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":141,"num_read_kb":529,"num_write":135,"num_write_kb":534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.452135+0000","last_change":"2024-09-21T21:32:07.212189+0000","last_active":"2024-09-21T21:33:15.452135+0000","last_peered":"2024-09-21T21:33:15.452135+0000","last_clean":"2024-09-21T21:33:15.452135+0000","last_became_active":"2024-09-21T21:32:05.304698+0000","last_became_peered":"2024-09-21T21:32:05.304698+0000","last_unstale":"2024-09-21T21:33:15.452135+0000","last_undegraded":"2024-09-21T21:33:15.452135+0000","last_fullsized":"2024-09-21T21:33:15.452135+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.450090+0000","last_change":"2024-09-21T21:32:07.205469+0000","last_active":"2024-09-21T21:33:15.450090+0000","last_peered":"2024-09-21T21:33:15.450090+0000","last_clean":"2024-09-21T21:33:15.450090+0000","last_became_active":"2024-09-21T21:32:05.258940+0000","last_became_peered":"2024-09-21T21:32:05.258940+0000","last_unstale":"2024-09-21T21:33:15.450090+0000","last_undegraded":"2024-09-21T21:33:15.450090+0000","last_fullsized":"2024-09-21T21:33:15.450090+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.451883+0000","last_change":"2024-09-21T21:32:07.213683+0000","last_active":"2024-09-21T21:33:15.451883+0000","last_peered":"2024-09-21T21:33:15.451883+0000","last_clean":"2024-09-21T21:33:15.451883+0000","last_became_active":"2024-09-21T21:32:05.219710+0000","last_became_peered":"2024-09-21T21:32:05.219710+0000","last_unstale":"2024-09-21T21:33:15.451883+0000","last_undegraded":"2024-09-21T21:33:15.451883+0000","last_fullsized":"2024-09-21T21:33:15.451883+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":99,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.446684+0000","last_change":"2024-09-21T21:32:03.188301+0000","last_active":"2024-09-21T21:33:15.446684+0000","last_peered":"2024-09-21T21:33:15.446684+0000","last_clean":"2024-09-21T21:33:15.446684+0000","last_became_active":"2024-09-21T21:32:03.168199+0000","last_became_peered":"2024-09-21T21:32:03.168199+0000","last_unstale":"2024-09-21T21:33:15.446684+0000","last_undegraded":"2024-09-21T21:33:15.446684+0000","last_fullsized":"2024-09-21T21:33:15.446684+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:02.098232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:02.098232+0000","last_clean_scrub_stamp":"2024-09-21T21:32:02.098232+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4131,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":324,"num_read_kb":549,"num_write":140,"num_write_kb":537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160654241792,"internally_reserved":0,"allocated":4378624,"data_stored":4378624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53550780416,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":468,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297260,"statfs":{"total":53552873472,"available":53552394240,"internally_reserved":0,"allocated":479232,"data_stored":479232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738468,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2044,"kb_used_data":2044,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295684,"statfs":{"total":53552873472,"available":53550780416,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705763,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1764,"kb_used_data":1764,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295964,"statfs":{"total":53552873472,"available":53551067136,"internally_reserved":0,"allocated":1806336,"data_stored":1806336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53550780416,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551067136,"internally_reserved":0,"allocated":1806336,"data_stored":1806336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550780416,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552394240,"internally_reserved":0,"allocated":479232,"data_stored":479232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T21:33:47.679 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T21:33:47.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:33:47.781+0000 7f5779791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:47.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:33:47.789+0000 7f5779791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:47.977 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:33:47.978 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":19,"fsid":"9c86c2d1-86e4-4d70-bab7-28b6d262ee29","created":"2024-09-21T21:31:53.391647+0000","modified":"2024-09-21T21:33:15.422468+0000","last_up_change":"2024-09-21T21:32:01.088108+0000","last_in_change":"2024-09-21T21:31:55.309162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T21:32:02.047960+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T21:32:03.242947+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"870c03e6-f121-4588-b0dc-6ce84b881ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3419716534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6805","nonce":3419716534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":3419716534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":3419716534}]},"public_addr":"172.21.15.129:6804/3419716534","cluster_addr":"172.21.15.129:6805/3419716534","heartbeat_back_addr":"172.21.15.129:6812/3419716534","heartbeat_front_addr":"172.21.15.129:6813/3419716534","state":["exists","up"]},{"osd":1,"uuid":"d2055726-f292-4619-94c7-cdd970552365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6803","nonce":1486109390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":1486109390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6807","nonce":1486109390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":1486109390}]},"public_addr":"172.21.15.129:6803/1486109390","cluster_addr":"172.21.15.129:6802/1486109390","heartbeat_back_addr":"172.21.15.129:6807/1486109390","heartbeat_front_addr":"172.21.15.129:6806/1486109390","state":["exists","up"]},{"osd":2,"uuid":"94e3907d-8e87-4ba8-bdd1-ad2eb5eecf12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":3156771813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":3156771813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":3156771813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6811","nonce":3156771813}]},"public_addr":"172.21.15.129:6809/3156771813","cluster_addr":"172.21.15.129:6808/3156771813","heartbeat_back_addr":"172.21.15.129:6810/3156771813","heartbeat_front_addr":"172.21.15.129:6811/3156771813","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:0/2282313657":"2024-09-21T22:33:14.735893+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T21:33:48.992 INFO:tasks.ceph:Scrubbing osd.0 2024-09-21T21:33:48.993 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-21T21:33:49.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:33:49.098+0000 7f5fcbac8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:49.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:33:49.110+0000 7f5fcbac8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:49.127 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T21:33:49.128 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "" 2024-09-21T21:33:49.128 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T21:33:49.139 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-21T21:33:49.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:33:49.246+0000 7f29a3a46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:49.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:33:49.255+0000 7f29a3a46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:49.457 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 0 to deep-scrub 2024-09-21T21:33:49.473 INFO:tasks.ceph:Scrubbing osd.1 2024-09-21T21:33:49.473 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-21T21:33:49.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:33:49.576+0000 7f5e4ef51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:49.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:33:49.587+0000 7f5e4ef51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:49.603 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T21:33:49.603 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "" 2024-09-21T21:33:49.604 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T21:33:49.615 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-21T21:33:49.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:33:49.715+0000 7fdf5bf91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:49.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:33:49.723+0000 7fdf5bf91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:49.913 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 1 to deep-scrub 2024-09-21T21:33:49.928 INFO:tasks.ceph:Scrubbing osd.2 2024-09-21T21:33:49.929 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-21T21:33:50.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:33:50.043+0000 7f742d791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:50.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:33:50.055+0000 7f742d791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:50.073 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T21:33:50.073 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "" 2024-09-21T21:33:50.073 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T21:33:50.085 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-21T21:33:50.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:33:50.186+0000 7fdcb8d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:50.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:33:50.194+0000 7fdcb8d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:50.386 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 2 to deep-scrub 2024-09-21T21:33:50.406 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T21:33:50.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:33:50.507+0000 7efe4f04f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:50.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:33:50.514+0000 7efe4f04f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:50.702 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:33:50.702 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-21T21:33:50.715 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2024-09-21T21:33:50.047803+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":465,"num_read_kb":601,"num_write":225,"num_write_kb":1646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":210,"ondisk_log_size":210,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":4276,"kb_used_data":4276,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156888908,"statfs":{"total":160658620416,"available":160654241792,"internally_reserved":0,"allocated":4378624,"data_stored":4378624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4112,"num_objects":-3,"num_object_clones":0,"num_object_copies":-6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":6,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002812"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.447208+0000","last_change":"2024-09-21T21:32:07.204573+0000","last_active":"2024-09-21T21:33:15.447208+0000","last_peered":"2024-09-21T21:33:15.447208+0000","last_clean":"2024-09-21T21:33:15.447208+0000","last_became_active":"2024-09-21T21:32:05.222859+0000","last_became_peered":"2024-09-21T21:32:05.222859+0000","last_unstale":"2024-09-21T21:33:15.447208+0000","last_undegraded":"2024-09-21T21:33:15.447208+0000","last_fullsized":"2024-09-21T21:33:15.447208+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.452562+0000","last_change":"2024-09-21T21:32:07.212812+0000","last_active":"2024-09-21T21:33:15.452562+0000","last_peered":"2024-09-21T21:33:15.452562+0000","last_clean":"2024-09-21T21:33:15.452562+0000","last_became_active":"2024-09-21T21:32:05.255580+0000","last_became_peered":"2024-09-21T21:32:05.255580+0000","last_unstale":"2024-09-21T21:33:15.452562+0000","last_undegraded":"2024-09-21T21:33:15.452562+0000","last_fullsized":"2024-09-21T21:33:15.452562+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"19'16","reported_seq":140,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:46.116076+0000","last_change":"2024-09-21T21:32:07.204008+0000","last_active":"2024-09-21T21:33:46.116076+0000","last_peered":"2024-09-21T21:33:46.116076+0000","last_clean":"2024-09-21T21:33:46.116076+0000","last_became_active":"2024-09-21T21:32:05.237491+0000","last_became_peered":"2024-09-21T21:32:05.237491+0000","last_unstale":"2024-09-21T21:33:46.116076+0000","last_undegraded":"2024-09-21T21:33:46.116076+0000","last_fullsized":"2024-09-21T21:33:46.116076+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":255,"num_read_kb":26,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"19'2","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:45.723150+0000","last_change":"2024-09-21T21:32:07.213553+0000","last_active":"2024-09-21T21:33:45.723150+0000","last_peered":"2024-09-21T21:33:45.723150+0000","last_clean":"2024-09-21T21:33:45.723150+0000","last_became_active":"2024-09-21T21:32:05.299045+0000","last_became_peered":"2024-09-21T21:32:05.299045+0000","last_unstale":"2024-09-21T21:33:45.723150+0000","last_undegraded":"2024-09-21T21:33:45.723150+0000","last_fullsized":"2024-09-21T21:33:45.723150+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'136","reported_seq":308,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:46.042290+0000","last_change":"2024-09-21T21:32:07.210460+0000","last_active":"2024-09-21T21:33:46.042290+0000","last_peered":"2024-09-21T21:33:46.042290+0000","last_clean":"2024-09-21T21:33:46.042290+0000","last_became_active":"2024-09-21T21:32:05.275105+0000","last_became_peered":"2024-09-21T21:32:05.275105+0000","last_unstale":"2024-09-21T21:33:46.042290+0000","last_undegraded":"2024-09-21T21:33:46.042290+0000","last_fullsized":"2024-09-21T21:33:46.042290+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":145,"num_read_kb":529,"num_write":136,"num_write_kb":534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.452135+0000","last_change":"2024-09-21T21:32:07.212189+0000","last_active":"2024-09-21T21:33:15.452135+0000","last_peered":"2024-09-21T21:33:15.452135+0000","last_clean":"2024-09-21T21:33:15.452135+0000","last_became_active":"2024-09-21T21:32:05.304698+0000","last_became_peered":"2024-09-21T21:32:05.304698+0000","last_unstale":"2024-09-21T21:33:15.452135+0000","last_undegraded":"2024-09-21T21:33:15.452135+0000","last_fullsized":"2024-09-21T21:33:15.452135+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.450090+0000","last_change":"2024-09-21T21:32:07.205469+0000","last_active":"2024-09-21T21:33:15.450090+0000","last_peered":"2024-09-21T21:33:15.450090+0000","last_clean":"2024-09-21T21:33:15.450090+0000","last_became_active":"2024-09-21T21:32:05.258940+0000","last_became_peered":"2024-09-21T21:32:05.258940+0000","last_unstale":"2024-09-21T21:33:15.450090+0000","last_undegraded":"2024-09-21T21:33:15.450090+0000","last_fullsized":"2024-09-21T21:33:15.450090+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"19'4","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:45.930716+0000","last_change":"2024-09-21T21:32:07.213683+0000","last_active":"2024-09-21T21:33:45.930716+0000","last_peered":"2024-09-21T21:33:45.930716+0000","last_clean":"2024-09-21T21:33:45.930716+0000","last_became_active":"2024-09-21T21:32:05.219710+0000","last_became_peered":"2024-09-21T21:32:05.219710+0000","last_unstale":"2024-09-21T21:33:45.930716+0000","last_undegraded":"2024-09-21T21:33:45.930716+0000","last_fullsized":"2024-09-21T21:33:45.930716+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:04.106240+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":99,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:15.446684+0000","last_change":"2024-09-21T21:32:03.188301+0000","last_active":"2024-09-21T21:33:15.446684+0000","last_peered":"2024-09-21T21:33:15.446684+0000","last_clean":"2024-09-21T21:33:15.446684+0000","last_became_active":"2024-09-21T21:32:03.168199+0000","last_became_peered":"2024-09-21T21:32:03.168199+0000","last_unstale":"2024-09-21T21:33:15.446684+0000","last_undegraded":"2024-09-21T21:33:15.446684+0000","last_fullsized":"2024-09-21T21:33:15.446684+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:02.098232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:02.098232+0000","last_clean_scrub_stamp":"2024-09-21T21:32:02.098232+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":412,"num_read_kb":555,"num_write":146,"num_write_kb":540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160654520320,"internally_reserved":0,"allocated":4100096,"data_stored":4100096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53550854144,"internally_reserved":0,"allocated":2019328,"data_stored":2019328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":468,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297260,"statfs":{"total":53552873472,"available":53552394240,"internally_reserved":0,"allocated":479232,"data_stored":479232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738473,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2044,"kb_used_data":2044,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295684,"statfs":{"total":53552873472,"available":53550780416,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705768,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1764,"kb_used_data":1764,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295964,"statfs":{"total":53552873472,"available":53551067136,"internally_reserved":0,"allocated":1806336,"data_stored":1806336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53550854144,"internally_reserved":0,"allocated":2019328,"data_stored":2019328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551337472,"internally_reserved":0,"allocated":1536000,"data_stored":1536000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550854144,"internally_reserved":0,"allocated":2019328,"data_stored":2019328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552328704,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T21:33:50.718 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-21T21:32:04.106240+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=21, tm_min=32, tm_sec=4, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=21, tm_min=33, tm_sec=47, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T21:33:50.718 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-21T21:32:04.106240+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=21, tm_min=32, tm_sec=4, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=21, tm_min=33, tm_sec=47, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T21:33:50.718 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-21T21:32:04.106240+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=21, tm_min=32, tm_sec=4, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=21, tm_min=33, tm_sec=47, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T21:33:50.718 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-21T21:32:04.106240+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=21, tm_min=32, tm_sec=4, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=21, tm_min=33, tm_sec=47, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T21:33:50.718 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-21T21:32:04.106240+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=21, tm_min=32, tm_sec=4, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=21, tm_min=33, tm_sec=47, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T21:33:50.719 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-21T21:32:04.106240+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=21, tm_min=32, tm_sec=4, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=21, tm_min=33, tm_sec=47, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T21:33:50.719 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-21T21:32:04.106240+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=21, tm_min=32, tm_sec=4, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=21, tm_min=33, tm_sec=47, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T21:33:50.719 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-21T21:32:04.106240+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=21, tm_min=32, tm_sec=4, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=21, tm_min=33, tm_sec=47, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T21:33:50.719 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-21T21:32:02.098232+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=21, tm_min=32, tm_sec=2, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=21, tm_min=33, tm_sec=47, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T21:33:50.719 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-21T21:34:10.720 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T21:34:10.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:34:10.835+0000 7f2fa5b11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:34:10.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:34:10.842+0000 7f2fa5b11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:34:11.029 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T21:34:11.030 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-21T21:34:11.042 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-09-21T21:34:10.052561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":466,"num_read_kb":601,"num_write":225,"num_write_kb":1646,"num_scrub_errors":0,"num_shallow_scrub_errors":2,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":216,"ondisk_log_size":216,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3876,"kb_used_data":3876,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889308,"statfs":{"total":160658620416,"available":160654651392,"internally_reserved":0,"allocated":3969024,"data_stored":3969024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002920"},"pg_stats":[{"pgid":"2.7","version":"19'1","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:49.919313+0000","last_change":"2024-09-21T21:33:49.958621+0000","last_active":"2024-09-21T21:33:49.919313+0000","last_peered":"2024-09-21T21:33:49.919313+0000","last_clean":"2024-09-21T21:33:49.919313+0000","last_became_active":"2024-09-21T21:32:05.222859+0000","last_became_peered":"2024-09-21T21:32:05.222859+0000","last_unstale":"2024-09-21T21:33:49.919313+0000","last_undegraded":"2024-09-21T21:33:49.919313+0000","last_fullsized":"2024-09-21T21:33:49.919313+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'1","last_scrub_stamp":"2024-09-21T21:33:49.958428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:33:49.958428+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"19'2","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:49.919638+0000","last_change":"2024-09-21T21:33:49.956137+0000","last_active":"2024-09-21T21:33:49.919638+0000","last_peered":"2024-09-21T21:33:49.919638+0000","last_clean":"2024-09-21T21:33:49.919638+0000","last_became_active":"2024-09-21T21:32:05.255580+0000","last_became_peered":"2024-09-21T21:32:05.255580+0000","last_unstale":"2024-09-21T21:33:49.919638+0000","last_undegraded":"2024-09-21T21:33:49.919638+0000","last_fullsized":"2024-09-21T21:33:49.919638+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2024-09-21T21:33:49.955861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:33:49.955861+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"19'17","reported_seq":142,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:49.917170+0000","last_change":"2024-09-21T21:33:49.945820+0000","last_active":"2024-09-21T21:33:49.917170+0000","last_peered":"2024-09-21T21:33:49.917170+0000","last_clean":"2024-09-21T21:33:49.917170+0000","last_became_active":"2024-09-21T21:32:05.237491+0000","last_became_peered":"2024-09-21T21:32:05.237491+0000","last_unstale":"2024-09-21T21:33:49.917170+0000","last_undegraded":"2024-09-21T21:33:49.917170+0000","last_fullsized":"2024-09-21T21:33:49.917170+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'17","last_scrub_stamp":"2024-09-21T21:33:49.945627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:33:49.945627+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":255,"num_read_kb":26,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"19'2","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:49.917168+0000","last_change":"2024-09-21T21:33:49.969625+0000","last_active":"2024-09-21T21:33:49.917168+0000","last_peered":"2024-09-21T21:33:49.917168+0000","last_clean":"2024-09-21T21:33:49.917168+0000","last_became_active":"2024-09-21T21:32:05.299045+0000","last_became_peered":"2024-09-21T21:32:05.299045+0000","last_unstale":"2024-09-21T21:33:49.917168+0000","last_undegraded":"2024-09-21T21:33:49.917168+0000","last_fullsized":"2024-09-21T21:33:49.917168+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2024-09-21T21:33:49.969304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:33:49.969304+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":1,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'136","reported_seq":310,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:49.458478+0000","last_change":"2024-09-21T21:33:49.480738+0000","last_active":"2024-09-21T21:33:49.458478+0000","last_peered":"2024-09-21T21:33:49.458478+0000","last_clean":"2024-09-21T21:33:49.458478+0000","last_became_active":"2024-09-21T21:32:05.275105+0000","last_became_peered":"2024-09-21T21:32:05.275105+0000","last_unstale":"2024-09-21T21:33:49.458478+0000","last_undegraded":"2024-09-21T21:33:49.458478+0000","last_fullsized":"2024-09-21T21:33:49.458478+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'136","last_scrub_stamp":"2024-09-21T21:33:49.480559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:33:49.480559+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":145,"num_read_kb":529,"num_write":136,"num_write_kb":534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:50.389962+0000","last_change":"2024-09-21T21:33:50.405799+0000","last_active":"2024-09-21T21:33:50.389962+0000","last_peered":"2024-09-21T21:33:50.389962+0000","last_clean":"2024-09-21T21:33:50.389962+0000","last_became_active":"2024-09-21T21:32:05.304698+0000","last_became_peered":"2024-09-21T21:32:05.304698+0000","last_unstale":"2024-09-21T21:33:50.389962+0000","last_undegraded":"2024-09-21T21:33:50.389962+0000","last_fullsized":"2024-09-21T21:33:50.389962+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:33:50.405625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:33:50.405625+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"19'1","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:50.387743+0000","last_change":"2024-09-21T21:33:50.402593+0000","last_active":"2024-09-21T21:33:50.387743+0000","last_peered":"2024-09-21T21:33:50.387743+0000","last_clean":"2024-09-21T21:33:50.387743+0000","last_became_active":"2024-09-21T21:32:05.258940+0000","last_became_peered":"2024-09-21T21:32:05.258940+0000","last_unstale":"2024-09-21T21:33:50.387743+0000","last_undegraded":"2024-09-21T21:33:50.387743+0000","last_fullsized":"2024-09-21T21:33:50.387743+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'1","last_scrub_stamp":"2024-09-21T21:33:50.402424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:33:50.402424+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"19'5","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:49.918096+0000","last_change":"2024-09-21T21:33:49.954438+0000","last_active":"2024-09-21T21:33:49.918096+0000","last_peered":"2024-09-21T21:33:49.918096+0000","last_clean":"2024-09-21T21:33:49.918096+0000","last_became_active":"2024-09-21T21:32:05.219710+0000","last_became_peered":"2024-09-21T21:32:05.219710+0000","last_unstale":"2024-09-21T21:33:49.918096+0000","last_undegraded":"2024-09-21T21:33:49.918096+0000","last_fullsized":"2024-09-21T21:33:49.918096+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'5","last_scrub_stamp":"2024-09-21T21:33:49.954141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:33:49.954141+0000","last_clean_scrub_stamp":"2024-09-21T21:32:04.106240+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":1,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":101,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T21:33:49.914771+0000","last_change":"2024-09-21T21:33:49.979014+0000","last_active":"2024-09-21T21:33:49.914771+0000","last_peered":"2024-09-21T21:33:49.914771+0000","last_clean":"2024-09-21T21:33:49.914771+0000","last_became_active":"2024-09-21T21:32:03.168199+0000","last_became_peered":"2024-09-21T21:32:03.168199+0000","last_unstale":"2024-09-21T21:33:49.914771+0000","last_undegraded":"2024-09-21T21:33:49.914771+0000","last_fullsized":"2024-09-21T21:33:49.914771+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"12'52","last_scrub_stamp":"2024-09-21T21:33:49.978785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:33:49.978785+0000","last_clean_scrub_stamp":"2024-09-21T21:32:02.098232+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":413,"num_read_kb":555,"num_write":146,"num_write_kb":540,"num_scrub_errors":0,"num_shallow_scrub_errors":2,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160654651392,"internally_reserved":0,"allocated":3969024,"data_stored":3969024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":164,"ondisk_log_size":164,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53550985216,"internally_reserved":0,"allocated":1888256,"data_stored":1888256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":532,"kb_used_data":532,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297196,"statfs":{"total":53552873472,"available":53552328704,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738493,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1844,"kb_used_data":1844,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295884,"statfs":{"total":53552873472,"available":53550985216,"internally_reserved":0,"allocated":1888256,"data_stored":1888256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705788,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1500,"kb_used_data":1500,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296228,"statfs":{"total":53552873472,"available":53551337472,"internally_reserved":0,"allocated":1536000,"data_stored":1536000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53550985216,"internally_reserved":0,"allocated":1888256,"data_stored":1888256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551337472,"internally_reserved":0,"allocated":1536000,"data_stored":1536000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550985216,"internally_reserved":0,"allocated":1888256,"data_stored":1888256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552328704,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T21:34:11.043 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-21T21:34:11.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:34:11.145+0000 7f428569c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:34:11.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T21:34:11.156+0000 7f428569c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:34:11.371 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-21T21:34:11.371 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-21T21:34:11.371 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-21T21:34:11.371 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T21:34:11.442 INFO:tasks.ceph.osd.0:Stopped 2024-09-21T21:34:11.442 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-21T21:34:11.442 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T21:34:11.520 INFO:tasks.ceph.osd.1:Stopped 2024-09-21T21:34:11.521 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-21T21:34:11.521 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T21:34:11.576 INFO:tasks.ceph.osd.2:Stopped 2024-09-21T21:34:11.576 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-21T21:34:11.576 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-21T21:34:11.576 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T21:34:11.606 INFO:tasks.ceph.mgr.x:Stopped 2024-09-21T21:34:11.606 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-21T21:34:11.606 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-21T21:34:11.606 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T21:34:11.657 INFO:tasks.ceph.mon.a:Stopped 2024-09-21T21:34:11.657 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-21T21:34:11.657 DEBUG:teuthology.orchestra.run.smithi129:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-21T21:34:11.727 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi129.front.sepia.ceph.com 2024-09-21T21:34:11.727 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-21T21:34:11.957 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi129.front.sepia.ceph.com 2024-09-21T21:34:11.958 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-21T21:34:12.037 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi129.front.sepia.ceph.com 2024-09-21T21:34:12.037 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-21T21:34:12.096 INFO:tasks.ceph:Archiving mon data... 2024-09-21T21:34:12.097 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914351/data/mon.a.tgz 2024-09-21T21:34:12.097 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2024-09-21T21:34:12.112 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.RERJ5zBbBL 2024-09-21T21:34:12.112 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.RERJ5zBbBL 2024-09-21T21:34:12.216 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.RERJ5zBbBL 2024-09-21T21:34:12.299 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.RERJ5zBbBL is 169KB 2024-09-21T21:34:12.351 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.RERJ5zBbBL 2024-09-21T21:34:12.365 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-21T21:34:12.366 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-21T21:34:12.486 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-21T21:34:12.486 INFO:tasks.ceph:Archiving crash dumps... 2024-09-21T21:34:12.488 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914351/remote/smithi129/crash 2024-09-21T21:34:12.489 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-21T21:34:12.549 INFO:tasks.ceph:Compressing logs... 2024-09-21T21:34:12.550 DEBUG:teuthology.orchestra.run.smithi129:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T21:34:12.613 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-21T21:34:12.614 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29211.log 2024-09-21T21:34:12.633 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28198.log 2024-09-21T21:34:12.633 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.29211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29744.log 2024-09-21T21:34:12.633 INFO:teuthology.orchestra.run.smithi129.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.29211.log.gz 2024-09-21T21:34:12.633 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.28198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28782.log 2024-09-21T21:34:12.634 INFO:teuthology.orchestra.run.smithi129.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28198.log.gz 2024-09-21T21:34:12.634 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.29744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29745.log 2024-09-21T21:34:12.634 INFO:teuthology.orchestra.run.smithi129.stderr: 47.3% -- replaced with /var/log/ceph/ceph-client.admin.29744.log.gz 2024-09-21T21:34:12.634 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.28782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30241.log 2024-09-21T21:34:12.634 INFO:teuthology.orchestra.run.smithi129.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28782.log.gz 2024-09-21T21:34:12.634 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29297.log 2024-09-21T21:34:12.634 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.29745.log: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.29745.log.gz 2024-09-21T21:34:12.634 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.30241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30157.log 2024-09-21T21:34:12.634 INFO:teuthology.orchestra.run.smithi129.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30241.log.gz 2024-09-21T21:34:12.634 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log/var/log/ceph/ceph-client.admin.29297.log: 2024-09-21T21:34:12.634 INFO:teuthology.orchestra.run.smithi129.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29297.log.gz 2024-09-21T21:34:12.634 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.30157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29210.log 2024-09-21T21:34:12.635 INFO:teuthology.orchestra.run.smithi129.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30157.log.gz 2024-09-21T21:34:12.635 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29083.log 2024-09-21T21:34:12.635 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.29210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30451.log 2024-09-21T21:34:12.635 INFO:teuthology.orchestra.run.smithi129.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29210.log.gz 2024-09-21T21:34:12.635 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.29083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28348.log 2024-09-21T21:34:12.635 INFO:teuthology.orchestra.run.smithi129.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29083.log.gz 2024-09-21T21:34:12.635 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.30451.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30451.log.gz 2024-09-21T21:34:12.635 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28961.log 2024-09-21T21:34:12.635 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.28348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29351.log 2024-09-21T21:34:12.635 INFO:teuthology.orchestra.run.smithi129.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28348.log.gz 2024-09-21T21:34:12.635 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.28961.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28961.log.gz 2024-09-21T21:34:12.635 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30283.log 2024-09-21T21:34:12.635 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.29351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29171.log 2024-09-21T21:34:12.635 INFO:teuthology.orchestra.run.smithi129.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29351.log.gz 2024-09-21T21:34:12.635 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.30283.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30283.log.gz 2024-09-21T21:34:12.636 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29423.log 2024-09-21T21:34:12.636 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-21T21:34:12.636 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.29171.log: /var/log/ceph/ceph-client.admin.29423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28826.log 2024-09-21T21:34:12.636 INFO:teuthology.orchestra.run.smithi129.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.29171.log.gz 2024-09-21T21:34:12.636 INFO:teuthology.orchestra.run.smithi129.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29423.log.gz 2024-09-21T21:34:12.636 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28726.log 2024-09-21T21:34:12.636 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.28826.log: 86.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-21T21:34:12.636 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28399.log 2024-09-21T21:34:12.636 INFO:teuthology.orchestra.run.smithi129.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.28826.log.gz 2024-09-21T21:34:12.636 INFO:teuthology.orchestra.run.smithi129.stderr:gzip/var/log/ceph/ceph-client.admin.28726.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30493.log 2024-09-21T21:34:12.636 INFO:teuthology.orchestra.run.smithi129.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28726.log.gz 2024-09-21T21:34:12.636 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.28399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30073.log 2024-09-21T21:34:12.636 INFO:teuthology.orchestra.run.smithi129.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28399.log.gz 2024-09-21T21:34:12.636 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.30493.log: gzip 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30493.log.gz -5 2024-09-21T21:34:12.636 INFO:teuthology.orchestra.run.smithi129.stderr: --verbose -- /var/log/ceph/ceph-client.admin.29085.log 2024-09-21T21:34:12.637 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.30073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29876.log 2024-09-21T21:34:12.637 INFO:teuthology.orchestra.run.smithi129.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30073.log.gz 2024-09-21T21:34:12.637 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.29085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30325.log 2024-09-21T21:34:12.637 INFO:teuthology.orchestra.run.smithi129.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29085.log.gz 2024-09-21T21:34:12.637 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.29876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29785.log 2024-09-21T21:34:12.637 INFO:teuthology.orchestra.run.smithi129.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29876.log.gz 2024-09-21T21:34:12.637 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.30325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29921.log 2024-09-21T21:34:12.637 INFO:teuthology.orchestra.run.smithi129.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30325.log.gz 2024-09-21T21:34:12.637 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.29785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29465.log 2024-09-21T21:34:12.637 INFO:teuthology.orchestra.run.smithi129.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.29785.log.gz 2024-09-21T21:34:12.637 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.29921.log: 66.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.29921.log.gz 2024-09-21T21:34:12.637 INFO:teuthology.orchestra.run.smithi129.stderr: -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-21T21:34:12.637 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.29465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30367.log 2024-09-21T21:34:12.637 INFO:teuthology.orchestra.run.smithi129.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29465.log.gz 2024-09-21T21:34:12.638 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28877.log 2024-09-21T21:34:12.638 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.30367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29727.log 2024-09-21T21:34:12.638 INFO:teuthology.orchestra.run.smithi129.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30367.log.gz 2024-09-21T21:34:12.638 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.28877.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-21T21:34:12.638 INFO:teuthology.orchestra.run.smithi129.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28877.log.gz 2024-09-21T21:34:12.638 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.29727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29807.log 2024-09-21T21:34:12.638 INFO:teuthology.orchestra.run.smithi129.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.29727.log.gz 2024-09-21T21:34:12.638 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29003.log 2024-09-21T21:34:12.638 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.29807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30199.log 2024-09-21T21:34:12.638 INFO:teuthology.orchestra.run.smithi129.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.29807.log.gz 2024-09-21T21:34:12.638 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.29003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29852.log 2024-09-21T21:34:12.638 INFO:teuthology.orchestra.run.smithi129.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29003.log.gz 2024-09-21T21:34:12.638 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.30199.log: gzip -5 --verbose 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30199.log.gz 2024-09-21T21:34:12.638 INFO:teuthology.orchestra.run.smithi129.stderr: -- /var/log/ceph/ceph-client.admin.30533.log 2024-09-21T21:34:12.639 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.29852.log: gzip -5 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29852.log.gz 2024-09-21T21:34:12.639 INFO:teuthology.orchestra.run.smithi129.stderr: --verbose -- /var/log/ceph/ceph.log 2024-09-21T21:34:12.639 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.30533.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30533.log.gz 2024-09-21T21:34:12.639 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29084.log 2024-09-21T21:34:12.639 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28308.log 2024-09-21T21:34:12.639 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.29084.log: gzip 50.0% -5 --verbose -- /var/log/ceph/ceph-client.admin.28587.log 2024-09-21T21:34:12.639 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29084.log.gz 85.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-21T21:34:12.639 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-21T21:34:12.639 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.28308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30031.log 2024-09-21T21:34:12.639 INFO:teuthology.orchestra.run.smithi129.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28308.log.gz 2024-09-21T21:34:12.639 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.28587.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-21T21:34:12.639 INFO:teuthology.orchestra.run.smithi129.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28587.log.gz 2024-09-21T21:34:12.639 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.30031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30115.log 2024-09-21T21:34:12.639 INFO:teuthology.orchestra.run.smithi129.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30031.log.gz 2024-09-21T21:34:12.640 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28919.log 2024-09-21T21:34:12.640 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.admin.30115.log: gzip -5 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30115.log.gz 2024-09-21T21:34:12.640 INFO:teuthology.orchestra.run.smithi129.stderr: --verbose -- /var/log/ceph/ceph-client.admin.29830.log 2024-09-21T21:34:12.640 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.28919.log: gzip 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28919.log.gz 2024-09-21T21:34:12.640 INFO:teuthology.orchestra.run.smithi129.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.28557.log 2024-09-21T21:34:12.640 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.29830.log: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.29830.log.gz 2024-09-21T21:34:12.640 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30409.log 2024-09-21T21:34:12.640 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5/var/log/ceph/ceph-client.admin.28557.log: --verbose -- /var/log/ceph/ceph-client.admin.29317.log 2024-09-21T21:34:12.640 INFO:teuthology.orchestra.run.smithi129.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28557.log.gz 2024-09-21T21:34:12.640 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.30409.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27162.log 2024-09-21T21:34:12.640 INFO:teuthology.orchestra.run.smithi129.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30409.log.gz 2024-09-21T21:34:12.640 INFO:teuthology.orchestra.run.smithi129.stderr:gzip/var/log/ceph/ceph-client.admin.29317.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.28110.log 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29317.log.gz 2024-09-21T21:34:12.640 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-21T21:34:12.640 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph.tmp-client.admin.27162.log: 66.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29507.log 2024-09-21T21:34:12.640 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph.tmp-client.admin.27162.log.gz 2024-09-21T21:34:12.641 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.28110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29900.log 2024-09-21T21:34:12.641 INFO:teuthology.orchestra.run.smithi129.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28110.log.gz 2024-09-21T21:34:12.641 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.29507.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29507.log.gz 2024-09-21T21:34:12.641 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.29900.log: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.29900.log.gz 2024-09-21T21:34:12.651 INFO:teuthology.orchestra.run.smithi129.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-21T21:34:12.671 INFO:teuthology.orchestra.run.smithi129.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-21T21:34:12.808 INFO:teuthology.orchestra.run.smithi129.stderr: 94.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-21T21:34:12.910 INFO:teuthology.orchestra.run.smithi129.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-21T21:34:12.924 INFO:teuthology.orchestra.run.smithi129.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-21T21:34:12.925 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-21T21:34:12.926 INFO:teuthology.orchestra.run.smithi129.stderr:real 0m0.322s 2024-09-21T21:34:12.926 INFO:teuthology.orchestra.run.smithi129.stderr:user 0m0.849s 2024-09-21T21:34:12.926 INFO:teuthology.orchestra.run.smithi129.stderr:sys 0m0.072s 2024-09-21T21:34:12.926 INFO:tasks.ceph:Archiving logs... 2024-09-21T21:34:12.926 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914351/remote/smithi129/log 2024-09-21T21:34:12.928 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T21:34:13.305 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-21T21:34:13.314 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-21T21:34:13.314 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T21:34:13.371 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-21T21:34:13.379 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-21T21:34:13.379 DEBUG:teuthology.orchestra.run.smithi129:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T21:34:13.400 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2024-09-21T21:34:13.404 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T21:34:13.404 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2024-09-21T21:34:13.404 INFO:teuthology.orchestra.run.smithi129.stdout:^* hv01.front.sepia.ceph.com 3 6 377 56 +248us[ +239us] +/- 39ms 2024-09-21T21:34:13.404 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 56 -829us[ -838us] +/- 56ms 2024-09-21T21:34:13.405 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 55 +553us[ +553us] +/- 73ms 2024-09-21T21:34:13.405 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T21:34:13.405 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-21T21:34:13.413 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-21T21:34:13.414 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-21T21:34:13.440 DEBUG:teuthology.orchestra.run.smithi129:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T21:34:13.506 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-21T21:34:13.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T21:34:13.534 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-21T21:34:13.543 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-21T21:34:13.575 INFO:teuthology.task.internal:Duration was 541.207847 seconds 2024-09-21T21:34:13.575 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-21T21:34:13.607 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-21T21:34:13.607 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T21:34:13.646 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T21:34:13.854 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-21T21:34:13.854 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi129.front.sepia.ceph.com 2024-09-21T21:34:13.854 DEBUG:teuthology.orchestra.run.smithi129:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T21:34:13.915 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-21T21:34:13.915 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T21:34:13.982 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-21T21:34:13.982 DEBUG:teuthology.orchestra.run.smithi129:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T21:34:14.120 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-21T21:34:14.130 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-21T21:34:14.130 DEBUG:teuthology.orchestra.run.smithi129:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T21:34:14.155 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-21T21:34:14.174 DEBUG:teuthology.orchestra.run.smithi129:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T21:34:14.218 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = core 2024-09-21T21:34:14.232 DEBUG:teuthology.orchestra.run.smithi129:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T21:34:14.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T21:34:14.287 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-21T21:34:14.296 INFO:teuthology.task.internal:Transferring archived files... 2024-09-21T21:34:14.297 DEBUG:teuthology.misc:Transferring archived files from smithi129:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914351/remote/smithi129 2024-09-21T21:34:14.297 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T21:34:14.360 INFO:teuthology.task.internal:Removing archive directory... 2024-09-21T21:34:14.360 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T21:34:14.411 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-21T21:34:14.421 INFO:teuthology.task.internal:Not uploading archives. 2024-09-21T21:34:14.421 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-21T21:34:14.458 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-21T21:34:14.458 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T21:34:14.473 INFO:teuthology.orchestra.run.smithi129.stdout: 265746 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 21:34 /home/ubuntu/cephtest 2024-09-21T21:34:14.474 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-21T21:34:14.533 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-21T21:34:14.609 INFO:teuthology.run:Summary data: description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_lock_and_fence} duration: 541.2078468799591 flavor: crimson owner: scheduled_teuthology@teuthology success: true 2024-09-21T21:34:14.610 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T21:34:14.691 INFO:teuthology.run:pass