2024-09-11T14:27:03.146 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-11T14:27:03.162 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T14:27:03.243 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901368 branch: main description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_python_api_tests_old_format} email: null first_in_suite: false job_id: '7901368' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd pool default crimson: true osd: crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug ms: 1 debug osd: 20 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore ompap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-1 osd.1: crimson seastar cpu cores: 2-3 osd.2: crimson seastar cpu cores: 4-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - \(SLOW_OPS\) - slow request sha1: 0d61375b8ab53be4c233f5b93e463edfdff9316a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: extra_system_packages: - python3-pytest flavor: crimson sha1: 0d61375b8ab53be4c233f5b93e463edfdff9316a workunit: branch: wip-matanb-crimson-seastore-logs env: CRIMSON_COMPAT: '1' sha1: dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 5813 sha1: 0d61375b8ab53be4c233f5b93e463edfdff9316a sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-matanb-crimson-seastore-logs suite_path: /home/teuthworker/src/github.com_matan-b_ceph_dbebf1a3db6db7b5579464ee715edcf1b3e9cce2/qa suite_relpath: qa suite_repo: https://github.com/matan-b/ceph.git suite_sha1: dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 targets: smithi188.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIpeqYZWsRy97+M4/mZVh8lxMGz6ycVQE3TYoZJrvY+0qZgaOiu4wdStLjwzIZzzioaLyYYgq8pKomtSa9o79FY= tasks: - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 flavor: crimson - workunit: clients: client.0: - rbd/test_librbd_python.sh -m 'not skip_if_crimson' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-11_13:56:22 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-11T14:27:03.243 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_matan-b_ceph_dbebf1a3db6db7b5579464ee715edcf1b3e9cce2/qa; will attempt to use it 2024-09-11T14:27:03.244 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_matan-b_ceph_dbebf1a3db6db7b5579464ee715edcf1b3e9cce2/qa/tasks 2024-09-11T14:27:03.244 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-11T14:27:03.245 INFO:teuthology.task.internal:Checking packages... 2024-09-11T14:27:03.267 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '0d61375b8ab53be4c233f5b93e463edfdff9316a' 2024-09-11T14:27:03.267 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-11T14:27:03.267 INFO:teuthology.packaging:ref: None 2024-09-11T14:27:03.268 INFO:teuthology.packaging:tag: None 2024-09-11T14:27:03.268 INFO:teuthology.packaging:branch: main 2024-09-11T14:27:03.268 INFO:teuthology.packaging:sha1: 0d61375b8ab53be4c233f5b93e463edfdff9316a 2024-09-11T14:27:03.268 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=main 2024-09-11T14:27:03.474 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4872.g0d61375b 2024-09-11T14:27:03.476 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-11T14:27:03.483 INFO:teuthology.task.internal:no buildpackages task found 2024-09-11T14:27:03.483 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-11T14:27:03.494 INFO:teuthology.task.internal:Saving configuration 2024-09-11T14:27:03.506 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-11T14:27:03.516 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-11T14:27:03.538 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901368', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 14:22:12.740670', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIpeqYZWsRy97+M4/mZVh8lxMGz6ycVQE3TYoZJrvY+0qZgaOiu4wdStLjwzIZzzioaLyYYgq8pKomtSa9o79FY='} 2024-09-11T14:27:03.538 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-11T14:27:03.551 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-11T14:27:03.551 INFO:teuthology.run_tasks:Running task console_log... 2024-09-11T14:27:03.585 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0777693ac0>, signals=[15]) 2024-09-11T14:27:03.585 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-11T14:27:03.593 INFO:teuthology.task.internal:Opening connections... 2024-09-11T14:27:03.593 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi188.front.sepia.ceph.com 2024-09-11T14:27:03.594 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T14:27:03.673 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-11T14:27:03.681 DEBUG:teuthology.orchestra.run.smithi188:> uname -m 2024-09-11T14:27:03.759 INFO:teuthology.orchestra.run.smithi188.stdout:x86_64 2024-09-11T14:27:03.760 DEBUG:teuthology.orchestra.run.smithi188:> cat /etc/os-release 2024-09-11T14:27:03.817 INFO:teuthology.orchestra.run.smithi188.stdout:NAME="CentOS Stream" 2024-09-11T14:27:03.817 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION="9" 2024-09-11T14:27:03.817 INFO:teuthology.orchestra.run.smithi188.stdout:ID="centos" 2024-09-11T14:27:03.817 INFO:teuthology.orchestra.run.smithi188.stdout:ID_LIKE="rhel fedora" 2024-09-11T14:27:03.818 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION_ID="9" 2024-09-11T14:27:03.818 INFO:teuthology.orchestra.run.smithi188.stdout:PLATFORM_ID="platform:el9" 2024-09-11T14:27:03.818 INFO:teuthology.orchestra.run.smithi188.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-11T14:27:03.818 INFO:teuthology.orchestra.run.smithi188.stdout:ANSI_COLOR="0;31" 2024-09-11T14:27:03.818 INFO:teuthology.orchestra.run.smithi188.stdout:LOGO="fedora-logo-icon" 2024-09-11T14:27:03.818 INFO:teuthology.orchestra.run.smithi188.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-11T14:27:03.818 INFO:teuthology.orchestra.run.smithi188.stdout:HOME_URL="https://centos.org/" 2024-09-11T14:27:03.818 INFO:teuthology.orchestra.run.smithi188.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-11T14:27:03.818 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-11T14:27:03.818 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-11T14:27:03.819 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server 2024-09-11T14:27:03.838 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-11T14:27:03.850 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-11T14:27:03.862 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-11T14:27:03.862 DEBUG:teuthology.orchestra.run.smithi188:> test '!' -e /home/ubuntu/cephtest 2024-09-11T14:27:03.879 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-11T14:27:03.886 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-11T14:27:03.886 DEBUG:teuthology.orchestra.run.smithi188:> test -z $(ls -A /var/lib/ceph) 2024-09-11T14:27:04.015 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-11T14:27:04.049 INFO:teuthology.run_tasks:Running task kernel... 2024-09-11T14:27:04.066 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-09-11T14:27:04.066 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-09-11T14:27:04.067 DEBUG:teuthology.orchestra.run.smithi188:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T14:27:04.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T14:27:04.086 DEBUG:teuthology.orchestra.run.smithi188:> uname -r 2024-09-11T14:27:04.143 INFO:teuthology.orchestra.run.smithi188.stdout:5.14.0-503.el9.x86_64 2024-09-11T14:27:04.143 INFO:teuthology.task.kernel:Running kernel on smithi188: 5.14.0-503.el9.x86_64 2024-09-11T14:27:04.143 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum install -y kernel 2024-09-11T14:27:06.624 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.2 MB 00:00 2024-09-11T14:27:10.449 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - AppStream 7.5 MB/s | 20 MB 00:02 2024-09-11T14:27:14.908 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - CRB 9.9 MB/s | 6.5 MB 00:00 2024-09-11T14:27:16.400 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - Extras packages 65 kB/s | 19 kB 00:00 2024-09-11T14:27:17.321 INFO:teuthology.orchestra.run.smithi188.stdout:Extra Packages for Enterprise Linux 28 MB/s | 23 MB 00:00 2024-09-11T14:27:22.842 INFO:teuthology.orchestra.run.smithi188.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-09-11T14:27:24.833 INFO:teuthology.orchestra.run.smithi188.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-11T14:27:24.833 INFO:teuthology.orchestra.run.smithi188.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-11T14:27:24.915 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-09-11T14:27:24.919 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-09-11T14:27:24.919 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-09-11T14:27:24.919 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-09-11T14:27:24.919 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-09-11T14:27:24.920 INFO:teuthology.orchestra.run.smithi188.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-11T14:27:24.920 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-09-11T14:27:24.920 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-11T14:27:24.920 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-11T14:27:24.920 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-11T14:27:24.920 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:27:24.920 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-09-11T14:27:24.920 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-09-11T14:27:24.920 INFO:teuthology.orchestra.run.smithi188.stdout:Install 4 Packages 2024-09-11T14:27:24.920 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:27:24.921 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 79 M 2024-09-11T14:27:24.921 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 126 M 2024-09-11T14:27:24.921 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-09-11T14:27:25.769 INFO:teuthology.orchestra.run.smithi188.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 38 kB/s | 23 kB 00:00 2024-09-11T14:27:26.720 INFO:teuthology.orchestra.run.smithi188.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2024-09-11T14:27:27.444 INFO:teuthology.orchestra.run.smithi188.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 15 MB/s | 35 MB 00:02 2024-09-11T14:27:28.170 INFO:teuthology.orchestra.run.smithi188.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 12 MB/s | 29 MB 00:02 2024-09-11T14:27:28.170 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:27:28.170 INFO:teuthology.orchestra.run.smithi188.stdout:Total 24 MB/s | 79 MB 00:03 2024-09-11T14:27:28.856 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-09-11T14:27:29.215 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-09-11T14:27:29.216 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-09-11T14:27:30.180 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-09-11T14:27:30.180 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-09-11T14:27:32.664 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-09-11T14:27:33.340 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-11T14:27:33.424 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T14:27:36.210 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T14:27:36.340 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T14:27:41.165 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T14:27:41.211 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:27:46.062 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:28:24.143 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:28:24.183 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:28:24.501 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:28:24.501 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-11T14:28:24.501 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T14:28:24.501 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T14:28:25.359 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:28:25.359 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:28:25.360 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-09-11T14:28:25.360 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-11T14:28:25.360 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-11T14:28:25.360 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-11T14:28:25.360 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-11T14:28:25.360 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:28:25.360 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-09-11T14:28:25.703 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T14:28:25.787 INFO:teuthology.orchestra.run.smithi188.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T14:28:25.787 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T14:28:25.788 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-11T14:28:25.788 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T14:28:25.788 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T14:28:25.788 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-11T14:28:25.788 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi188.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T14:28:25.788 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T14:28:25.847 INFO:teuthology.orchestra.run.smithi188.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T14:28:25.847 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T14:28:25.848 DEBUG:teuthology.orchestra.run.smithi188:> sudo rpm -qi grub2-tools 2024-09-11T14:28:25.885 INFO:teuthology.orchestra.run.smithi188.stdout:Name : grub2-tools 2024-09-11T14:28:25.885 INFO:teuthology.orchestra.run.smithi188.stdout:Epoch : 1 2024-09-11T14:28:25.885 INFO:teuthology.orchestra.run.smithi188.stdout:Version : 2.06 2024-09-11T14:28:25.886 INFO:teuthology.orchestra.run.smithi188.stdout:Release : 82.el9 2024-09-11T14:28:25.886 INFO:teuthology.orchestra.run.smithi188.stdout:Architecture: x86_64 2024-09-11T14:28:25.886 INFO:teuthology.orchestra.run.smithi188.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-11T14:28:25.886 INFO:teuthology.orchestra.run.smithi188.stdout:Group : Unspecified 2024-09-11T14:28:25.886 INFO:teuthology.orchestra.run.smithi188.stdout:Size : 8274670 2024-09-11T14:28:25.886 INFO:teuthology.orchestra.run.smithi188.stdout:License : GPLv3+ 2024-09-11T14:28:25.886 INFO:teuthology.orchestra.run.smithi188.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-11T14:28:25.886 INFO:teuthology.orchestra.run.smithi188.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-11T14:28:25.886 INFO:teuthology.orchestra.run.smithi188.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-11T14:28:25.886 INFO:teuthology.orchestra.run.smithi188.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-11T14:28:25.886 INFO:teuthology.orchestra.run.smithi188.stdout:Packager : builder@centos.org 2024-09-11T14:28:25.886 INFO:teuthology.orchestra.run.smithi188.stdout:Vendor : CentOS 2024-09-11T14:28:25.887 INFO:teuthology.orchestra.run.smithi188.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-11T14:28:25.887 INFO:teuthology.orchestra.run.smithi188.stdout:Summary : Support tools for GRUB. 2024-09-11T14:28:25.887 INFO:teuthology.orchestra.run.smithi188.stdout:Description : 2024-09-11T14:28:25.887 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:28:25.887 INFO:teuthology.orchestra.run.smithi188.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-11T14:28:25.887 INFO:teuthology.orchestra.run.smithi188.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-11T14:28:25.887 INFO:teuthology.orchestra.run.smithi188.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-11T14:28:25.887 INFO:teuthology.orchestra.run.smithi188.stdout:hardware devices. 2024-09-11T14:28:25.887 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:28:25.887 INFO:teuthology.orchestra.run.smithi188.stdout:This subpackage provides tools for support of all platforms. 2024-09-11T14:28:25.888 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-11T14:28:25.888 INFO:teuthology.task.kernel:Updating grub on smithi188 to boot 5.14.0-505.el9.x86_64 2024-09-11T14:28:25.888 DEBUG:teuthology.orchestra.run.smithi188:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-11T14:28:26.500 INFO:teuthology.orchestra.run.smithi188.stderr:Generating grub configuration file ... 2024-09-11T14:28:27.854 INFO:teuthology.orchestra.run.smithi188.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-11T14:28:27.889 INFO:teuthology.orchestra.run.smithi188.stderr:done 2024-09-11T14:28:27.892 DEBUG:teuthology.orchestra.run.smithi188:> mktemp 2024-09-11T14:28:27.907 INFO:teuthology.orchestra.run.smithi188.stdout:/tmp/tmp.9f6xhye9gX 2024-09-11T14:28:27.908 DEBUG:teuthology.orchestra.run.smithi188:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.9f6xhye9gX 2024-09-11T14:28:27.974 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 0666 /tmp/tmp.9f6xhye9gX 2024-09-11T14:28:28.107 DEBUG:teuthology.orchestra.remote:smithi188:/tmp/tmp.9f6xhye9gX is 6KB 2024-09-11T14:28:28.157 DEBUG:teuthology.orchestra.run.smithi188:> rm -fr /tmp/tmp.9f6xhye9gX 2024-09-11T14:28:28.172 DEBUG:teuthology.orchestra.run.smithi188:> sudo /bin/ls /boot/loader/entries || true 2024-09-11T14:28:28.234 INFO:teuthology.orchestra.run.smithi188.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-11T14:28:28.235 INFO:teuthology.orchestra.run.smithi188.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-11T14:28:28.235 INFO:teuthology.orchestra.run.smithi188.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-11T14:28:28.235 INFO:teuthology.orchestra.run.smithi188.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-11T14:28:28.236 DEBUG:teuthology.orchestra.run.smithi188:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-11T14:28:28.395 DEBUG:teuthology.orchestra.run.smithi188:> sudo shutdown -r now 2024-09-11T14:28:58.425 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-11T14:28:58.426 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-09-11T14:28:58.427 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T14:29:58.490 DEBUG:teuthology.orchestra.remote:timed out 2024-09-11T14:30:07.499 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-09-11T14:30:07.501 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T14:30:08.039 DEBUG:teuthology.orchestra.run.smithi188:> true 2024-09-11T14:30:08.422 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-09-11T14:30:08.422 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-11T14:30:08.422 DEBUG:teuthology.orchestra.run.smithi188:> uname -r 2024-09-11T14:30:08.436 INFO:teuthology.orchestra.run.smithi188.stdout:5.14.0-505.el9.x86_64 2024-09-11T14:30:08.436 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-11T14:30:08.437 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T14:30:08.437 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-11T14:30:09.438 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-11T14:30:09.438 DEBUG:teuthology.orchestra.run.smithi188:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T14:30:09.543 INFO:teuthology.orchestra.run.smithi188.stdout:ttyS1 2024-09-11T14:30:09.570 DEBUG:teuthology.parallel:result is None 2024-09-11T14:30:09.570 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-11T14:30:09.578 INFO:teuthology.task.internal:Creating test directory... 2024-09-11T14:30:09.578 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T14:30:09.593 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-11T14:30:09.625 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-11T14:30:09.637 INFO:teuthology.task.internal:Creating archive directory... 2024-09-11T14:30:09.638 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T14:30:09.658 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-11T14:30:09.669 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-11T14:30:09.669 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T14:30:09.722 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T14:30:09.732 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T14:30:09.733 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-11T14:30:09.744 INFO:teuthology.task.internal:Configuring sudo... 2024-09-11T14:30:09.744 DEBUG:teuthology.orchestra.run.smithi188:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T14:30:09.794 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-11T14:30:09.808 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-11T14:30:09.808 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T14:30:09.846 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T14:30:09.928 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T14:30:09.973 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T14:30:10.045 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T14:30:10.088 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-11T14:30:10.088 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T14:30:10.150 DEBUG:teuthology.orchestra.run.smithi188:> sudo service rsyslog restart 2024-09-11T14:30:10.248 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T14:30:10.582 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-11T14:30:10.590 INFO:teuthology.task.internal:Starting timer... 2024-09-11T14:30:10.590 INFO:teuthology.run_tasks:Running task pcp... 2024-09-11T14:30:10.607 INFO:teuthology.run_tasks:Running task selinux... 2024-09-11T14:30:10.619 DEBUG:teuthology.orchestra.run.smithi188:> sudo service auditd rotate 2024-09-11T14:30:10.666 INFO:teuthology.orchestra.run.smithi188.stdout:Rotating logs: 2024-09-11T14:30:10.668 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-11T14:30:10.669 DEBUG:teuthology.orchestra.run.smithi188:> /usr/sbin/getenforce 2024-09-11T14:30:10.688 INFO:teuthology.orchestra.run.smithi188.stdout:Permissive 2024-09-11T14:30:10.688 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi188.front.sepia.ceph.com': 'permissive'} 2024-09-11T14:30:10.688 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T14:30:10.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T14:30:10.753 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-11T14:30:10.753 DEBUG:teuthology.orchestra.run.smithi188:> sudo /usr/sbin/setenforce permissive 2024-09-11T14:30:10.825 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-11T14:30:10.835 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-11T14:30:10.837 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-11T14:30:10.948 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-11T14:30:10.960 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-11T14:30:10.961 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi188.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-11T14:33:48.869 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi188.front.sepia.ceph.com')] 2024-09-11T14:33:48.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-09-11T14:33:48.872 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T14:33:48.948 DEBUG:teuthology.orchestra.run.smithi188:> true 2024-09-11T14:33:49.032 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-09-11T14:33:49.032 INFO:teuthology.run_tasks:Running task clock... 2024-09-11T14:33:49.042 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-11T14:33:49.042 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T14:33:49.043 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T14:33:49.101 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-11T14:33:49.116 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-11T14:33:49.141 INFO:teuthology.orchestra.run.smithi188.stderr:sudo: ntpd: command not found 2024-09-11T14:33:49.152 INFO:teuthology.orchestra.run.smithi188.stdout:506 Cannot talk to daemon 2024-09-11T14:33:49.166 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-11T14:33:49.180 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-11T14:33:49.235 INFO:teuthology.orchestra.run.smithi188.stderr:bash: line 1: ntpq: command not found 2024-09-11T14:33:49.238 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T14:33:49.238 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2024-09-11T14:33:49.239 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T14:33:49.239 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T14:33:49.239 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T14:33:49.239 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T14:33:49.240 INFO:teuthology.run_tasks:Running task install... 2024-09-11T14:33:49.248 DEBUG:teuthology.task.install:project ceph 2024-09-11T14:33:49.248 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'extra_system_packages': ['python3-pytest'], 'flavor': 'crimson', 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a'}} 2024-09-11T14:33:49.248 DEBUG:teuthology.task.install:config {'extra_system_packages': ['python3-pytest'], 'flavor': 'crimson', 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a'} 2024-09-11T14:33:49.248 INFO:teuthology.task.install:Using flavor: crimson 2024-09-11T14:33:49.254 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-11T14:33:49.254 INFO:teuthology.task.install:extra packages: [] 2024-09-11T14:33:49.254 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': ['python3-pytest'], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a', 'tag': None, 'wait_for_package': False} 2024-09-11T14:33:49.255 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=0d61375b8ab53be4c233f5b93e463edfdff9316a 2024-09-11T14:33:49.403 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/main/0d61375b8ab53be4c233f5b93e463edfdff9316a/centos/9/flavors/crimson/ 2024-09-11T14:33:49.403 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4872.g0d61375b 2024-09-11T14:33:49.517 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/main/0d61375b8ab53be4c233f5b93e463edfdff9316a/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/main/0d61375b8ab53be4c233f5b93e463edfdff9316a/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/main/0d61375b8ab53be4c233f5b93e463edfdff9316a/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-11T14:33:49.517 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-11T14:33:49.517 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-11T14:33:49.543 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-pytest, python3-pytest on remote rpm x86_64 2024-09-11T14:33:49.543 DEBUG:teuthology.orchestra.run.smithi188:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0d61375b8ab53be4c233f5b93e463edfdff9316a/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-11T14:33:49.609 DEBUG:teuthology.orchestra.run.smithi188:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-11T14:33:49.684 DEBUG:teuthology.orchestra.run.smithi188:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-11T14:33:49.748 INFO:teuthology.orchestra.run.smithi188.stdout:check_obsoletes = 1 2024-09-11T14:33:49.750 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum clean all 2024-09-11T14:33:49.985 INFO:teuthology.orchestra.run.smithi188.stdout:45 files removed 2024-09-11T14:33:50.012 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-pytest python3-pytest 2024-09-11T14:33:50.572 INFO:teuthology.orchestra.run.smithi188.stdout:ceph packages for x86_64 322 kB/s | 89 kB 00:00 2024-09-11T14:33:50.856 INFO:teuthology.orchestra.run.smithi188.stdout:ceph noarch packages 101 kB/s | 24 kB 00:00 2024-09-11T14:33:51.123 INFO:teuthology.orchestra.run.smithi188.stdout:ceph source packages 9.1 kB/s | 2.1 kB 00:00 2024-09-11T14:33:51.673 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.2 MB 00:00 2024-09-11T14:33:56.141 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - AppStream 6.1 MB/s | 20 MB 00:03 2024-09-11T14:34:01.250 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - CRB 5.1 MB/s | 6.5 MB 00:01 2024-09-11T14:34:02.717 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - Extras packages 77 kB/s | 19 kB 00:00 2024-09-11T14:34:03.796 INFO:teuthology.orchestra.run.smithi188.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-11T14:34:09.235 INFO:teuthology.orchestra.run.smithi188.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-09-11T14:34:11.589 INFO:teuthology.orchestra.run.smithi188.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-11T14:34:11.590 INFO:teuthology.orchestra.run.smithi188.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-11T14:34:11.652 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-09-11T14:34:11.670 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================= 2024-09-11T14:34:11.670 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-09-11T14:34:11.670 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================= 2024-09-11T14:34:11.670 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-09-11T14:34:11.670 INFO:teuthology.orchestra.run.smithi188.stdout: ceph x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 6.5 k 2024-09-11T14:34:11.670 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 6.1 M 2024-09-11T14:34:11.670 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 884 k 2024-09-11T14:34:11.670 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 158 k 2024-09-11T14:34:11.670 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 1.6 M 2024-09-11T14:34:11.670 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 154 k 2024-09-11T14:34:11.670 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 14 M 2024-09-11T14:34:11.671 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 7.4 M 2024-09-11T14:34:11.671 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 49 k 2024-09-11T14:34:11.671 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 10 M 2024-09-11T14:34:11.671 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 85 M 2024-09-11T14:34:11.671 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-volume noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 285 k 2024-09-11T14:34:11.671 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 784 k 2024-09-11T14:34:11.671 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 32 k 2024-09-11T14:34:11.671 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 808 k 2024-09-11T14:34:11.671 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 125 k 2024-09-11T14:34:11.671 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 159 k 2024-09-11T14:34:11.671 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytest noarch 6.2.2-7.el9 appstream 519 k 2024-09-11T14:34:11.671 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 316 k 2024-09-11T14:34:11.671 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 303 k 2024-09-11T14:34:11.672 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 99 k 2024-09-11T14:34:11.672 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 91 k 2024-09-11T14:34:11.672 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 3.5 M 2024-09-11T14:34:11.672 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 177 k 2024-09-11T14:34:11.672 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrading: 2024-09-11T14:34:11.672 INFO:teuthology.orchestra.run.smithi188.stdout: librados2 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 3.5 M 2024-09-11T14:34:11.672 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 3.8 M 2024-09-11T14:34:11.672 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-09-11T14:34:11.672 INFO:teuthology.orchestra.run.smithi188.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-11T14:34:11.672 INFO:teuthology.orchestra.run.smithi188.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-11T14:34:11.672 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 20 M 2024-09-11T14:34:11.673 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 34 k 2024-09-11T14:34:11.673 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 2.2 M 2024-09-11T14:34:11.673 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 291 k 2024-09-11T14:34:11.673 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 5.1 M 2024-09-11T14:34:11.673 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 71 M 2024-09-11T14:34:11.673 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 16 k 2024-09-11T14:34:11.673 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 25 k 2024-09-11T14:34:11.673 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-11T14:34:11.673 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-11T14:34:11.673 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-11T14:34:11.673 INFO:teuthology.orchestra.run.smithi188.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-11T14:34:11.673 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-11T14:34:11.673 INFO:teuthology.orchestra.run.smithi188.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-11T14:34:11.674 INFO:teuthology.orchestra.run.smithi188.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-11T14:34:11.674 INFO:teuthology.orchestra.run.smithi188.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-11T14:34:11.674 INFO:teuthology.orchestra.run.smithi188.stdout: libasan8 x86_64 14.2.1-1.2.el9 appstream 513 k 2024-09-11T14:34:11.674 INFO:teuthology.orchestra.run.smithi188.stdout: libcephsqlite x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 175 k 2024-09-11T14:34:11.674 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-11T14:34:11.674 INFO:teuthology.orchestra.run.smithi188.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-11T14:34:11.674 INFO:teuthology.orchestra.run.smithi188.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-11T14:34:11.674 INFO:teuthology.orchestra.run.smithi188.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-11T14:34:11.674 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-11T14:34:11.674 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-11T14:34:11.674 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 558 k 2024-09-11T14:34:11.674 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-11T14:34:11.674 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 5.4 M 2024-09-11T14:34:11.675 INFO:teuthology.orchestra.run.smithi188.stdout: libubsan x86_64 11.5.0-2.el9 appstream 183 k 2024-09-11T14:34:11.675 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-11T14:34:11.675 INFO:teuthology.orchestra.run.smithi188.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-09-11T14:34:11.675 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-11T14:34:11.675 INFO:teuthology.orchestra.run.smithi188.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-11T14:34:11.675 INFO:teuthology.orchestra.run.smithi188.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-11T14:34:11.675 INFO:teuthology.orchestra.run.smithi188.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-11T14:34:11.675 INFO:teuthology.orchestra.run.smithi188.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-11T14:34:11.675 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-11T14:34:11.675 INFO:teuthology.orchestra.run.smithi188.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-11T14:34:11.675 INFO:teuthology.orchestra.run.smithi188.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-11T14:34:11.675 INFO:teuthology.orchestra.run.smithi188.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-11T14:34:11.675 INFO:teuthology.orchestra.run.smithi188.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-11T14:34:11.675 INFO:teuthology.orchestra.run.smithi188.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-11T14:34:11.675 INFO:teuthology.orchestra.run.smithi188.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-11T14:34:11.676 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-11T14:34:11.676 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-11T14:34:11.676 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 45 k 2024-09-11T14:34:11.676 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 143 k 2024-09-11T14:34:11.676 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-11T14:34:11.676 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-11T14:34:11.676 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-11T14:34:11.676 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-11T14:34:11.676 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-11T14:34:11.676 INFO:teuthology.orchestra.run.smithi188.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-11T14:34:11.676 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-11T14:34:11.676 INFO:teuthology.orchestra.run.smithi188.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-11T14:34:11.676 INFO:teuthology.orchestra.run.smithi188.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-11T14:34:11.676 INFO:teuthology.orchestra.run.smithi188.stdout: python3-iniconfig noarch 1.1.1-7.el9 appstream 17 k 2024-09-11T14:34:11.677 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-11T14:34:11.677 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-11T14:34:11.677 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-11T14:34:11.677 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-11T14:34:11.677 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-11T14:34:11.677 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-11T14:34:11.677 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-11T14:34:11.677 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-11T14:34:11.677 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-11T14:34:11.677 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-11T14:34:11.677 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-11T14:34:11.677 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-11T14:34:11.677 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-11T14:34:11.677 INFO:teuthology.orchestra.run.smithi188.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-11T14:34:11.678 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-11T14:34:11.678 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-11T14:34:11.678 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-11T14:34:11.678 INFO:teuthology.orchestra.run.smithi188.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-11T14:34:11.678 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-11T14:34:11.678 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pluggy noarch 0.13.1-7.el9 appstream 41 k 2024-09-11T14:34:11.678 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-11T14:34:11.678 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-11T14:34:11.678 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-11T14:34:11.678 INFO:teuthology.orchestra.run.smithi188.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-11T14:34:11.678 INFO:teuthology.orchestra.run.smithi188.stdout: python3-py noarch 1.10.0-6.el9 appstream 477 k 2024-09-11T14:34:11.678 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-11T14:34:11.678 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-11T14:34:11.678 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-11T14:34:11.678 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-11T14:34:11.679 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-11T14:34:11.679 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-11T14:34:11.679 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-11T14:34:11.679 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-11T14:34:11.679 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-11T14:34:11.679 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-11T14:34:11.679 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-11T14:34:11.679 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-11T14:34:11.679 INFO:teuthology.orchestra.run.smithi188.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-11T14:34:11.679 INFO:teuthology.orchestra.run.smithi188.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-11T14:34:11.679 INFO:teuthology.orchestra.run.smithi188.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-11T14:34:11.679 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-11T14:34:11.679 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-11T14:34:11.679 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-11T14:34:11.680 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-11T14:34:11.680 INFO:teuthology.orchestra.run.smithi188.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-11T14:34:11.680 INFO:teuthology.orchestra.run.smithi188.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-11T14:34:11.680 INFO:teuthology.orchestra.run.smithi188.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-11T14:34:11.680 INFO:teuthology.orchestra.run.smithi188.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-11T14:34:11.680 INFO:teuthology.orchestra.run.smithi188.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-11T14:34:11.680 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-11T14:34:11.680 INFO:teuthology.orchestra.run.smithi188.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-09-11T14:34:11.680 INFO:teuthology.orchestra.run.smithi188.stdout:Installing weak dependencies: 2024-09-11T14:34:11.680 INFO:teuthology.orchestra.run.smithi188.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-11T14:34:11.680 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:34:11.680 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-09-11T14:34:11.680 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================= 2024-09-11T14:34:11.681 INFO:teuthology.orchestra.run.smithi188.stdout:Install 134 Packages 2024-09-11T14:34:11.681 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrade 2 Packages 2024-09-11T14:34:11.681 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:34:11.681 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 305 M 2024-09-11T14:34:11.681 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-09-11T14:34:12.536 INFO:teuthology.orchestra.run.smithi188.stdout:(1/136): ceph-19.3.0-4872.g0d61375b.el9.x86_64. 51 kB/s | 6.5 kB 00:00 2024-09-11T14:34:12.745 INFO:teuthology.orchestra.run.smithi188.stdout:(2/136): ceph-fuse-19.3.0-4872.g0d61375b.el9.x8 4.1 MB/s | 884 kB 00:00 2024-09-11T14:34:12.828 INFO:teuthology.orchestra.run.smithi188.stdout:(3/136): ceph-immutable-object-cache-19.3.0-487 1.9 MB/s | 158 kB 00:00 2024-09-11T14:34:12.904 INFO:teuthology.orchestra.run.smithi188.stdout:(4/136): ceph-base-19.3.0-4872.g0d61375b.el9.x8 12 MB/s | 6.1 MB 00:00 2024-09-11T14:34:12.995 INFO:teuthology.orchestra.run.smithi188.stdout:(5/136): ceph-mgr-19.3.0-4872.g0d61375b.el9.x86 17 MB/s | 1.6 MB 00:00 2024-09-11T14:34:13.271 INFO:teuthology.orchestra.run.smithi188.stdout:(6/136): ceph-mon-19.3.0-4872.g0d61375b.el9.x86 19 MB/s | 5.1 MB 00:00 2024-09-11T14:34:13.579 INFO:teuthology.orchestra.run.smithi188.stdout:(7/136): ceph-common-19.3.0-4872.g0d61375b.el9. 17 MB/s | 20 MB 00:01 2024-09-11T14:34:13.638 INFO:teuthology.orchestra.run.smithi188.stdout:(8/136): ceph-mds-19.3.0-4872.g0d61375b.el9.x86 2.7 MB/s | 2.2 MB 00:00 2024-09-11T14:34:13.697 INFO:teuthology.orchestra.run.smithi188.stdout:(9/136): ceph-selinux-19.3.0-4872.g0d61375b.el9 430 kB/s | 25 kB 00:00 2024-09-11T14:34:14.131 INFO:teuthology.orchestra.run.smithi188.stdout:(10/136): ceph-radosgw-19.3.0-4872.g0d61375b.el 19 MB/s | 10 MB 00:00 2024-09-11T14:34:14.172 INFO:teuthology.orchestra.run.smithi188.stdout:(11/136): libcephfs-devel-19.3.0-4872.g0d61375b 766 kB/s | 32 kB 00:00 2024-09-11T14:34:14.256 INFO:teuthology.orchestra.run.smithi188.stdout:(12/136): libcephfs2-19.3.0-4872.g0d61375b.el9. 9.5 MB/s | 808 kB 00:00 2024-09-11T14:34:14.323 INFO:teuthology.orchestra.run.smithi188.stdout:(13/136): libcephsqlite-19.3.0-4872.g0d61375b.e 2.6 MB/s | 175 kB 00:00 2024-09-11T14:34:14.390 INFO:teuthology.orchestra.run.smithi188.stdout:(14/136): librados-devel-19.3.0-4872.g0d61375b. 1.8 MB/s | 125 kB 00:00 2024-09-11T14:34:14.465 INFO:teuthology.orchestra.run.smithi188.stdout:(15/136): libradosstriper1-19.3.0-4872.g0d61375 7.3 MB/s | 558 kB 00:00 2024-09-11T14:34:14.799 INFO:teuthology.orchestra.run.smithi188.stdout:(16/136): librgw2-19.3.0-4872.g0d61375b.el9.x86 16 MB/s | 5.4 MB 00:00 2024-09-11T14:34:14.849 INFO:teuthology.orchestra.run.smithi188.stdout:(17/136): python3-ceph-argparse-19.3.0-4872.g0d 905 kB/s | 45 kB 00:00 2024-09-11T14:34:14.925 INFO:teuthology.orchestra.run.smithi188.stdout:(18/136): python3-ceph-common-19.3.0-4872.g0d61 1.9 MB/s | 143 kB 00:00 2024-09-11T14:34:14.999 INFO:teuthology.orchestra.run.smithi188.stdout:(19/136): python3-cephfs-19.3.0-4872.g0d61375b. 2.1 MB/s | 159 kB 00:00 2024-09-11T14:34:15.075 INFO:teuthology.orchestra.run.smithi188.stdout:(20/136): python3-rados-19.3.0-4872.g0d61375b.e 4.1 MB/s | 316 kB 00:00 2024-09-11T14:34:15.150 INFO:teuthology.orchestra.run.smithi188.stdout:(21/136): python3-rbd-19.3.0-4872.g0d61375b.el9 3.9 MB/s | 303 kB 00:00 2024-09-11T14:34:15.225 INFO:teuthology.orchestra.run.smithi188.stdout:(22/136): python3-rgw-19.3.0-4872.g0d61375b.el9 1.3 MB/s | 99 kB 00:00 2024-09-11T14:34:15.301 INFO:teuthology.orchestra.run.smithi188.stdout:(23/136): rbd-fuse-19.3.0-4872.g0d61375b.el9.x8 1.2 MB/s | 91 kB 00:00 2024-09-11T14:34:16.068 INFO:teuthology.orchestra.run.smithi188.stdout:(24/136): ceph-osd-19.3.0-4872.g0d61375b.el9.x8 25 MB/s | 71 MB 00:02 2024-09-11T14:34:16.126 INFO:teuthology.orchestra.run.smithi188.stdout:(25/136): rbd-mirror-19.3.0-4872.g0d61375b.el9. 4.2 MB/s | 3.5 MB 00:00 2024-09-11T14:34:16.152 INFO:teuthology.orchestra.run.smithi188.stdout:(26/136): rbd-nbd-19.3.0-4872.g0d61375b.el9.x86 2.1 MB/s | 177 kB 00:00 2024-09-11T14:34:16.177 INFO:teuthology.orchestra.run.smithi188.stdout:(27/136): ceph-grafana-dashboards-19.3.0-4872.g 682 kB/s | 34 kB 00:00 2024-09-11T14:34:16.211 INFO:teuthology.orchestra.run.smithi188.stdout:(28/136): ceph-mgr-cephadm-19.3.0-4872.g0d61375 2.6 MB/s | 154 kB 00:00 2024-09-11T14:34:16.628 INFO:teuthology.orchestra.run.smithi188.stdout:(29/136): ceph-mgr-diskprediction-local-19.3.0- 18 MB/s | 7.4 MB 00:00 2024-09-11T14:34:16.686 INFO:teuthology.orchestra.run.smithi188.stdout:(30/136): ceph-mgr-modules-core-19.3.0-4872.g0d 4.9 MB/s | 291 kB 00:00 2024-09-11T14:34:16.828 INFO:teuthology.orchestra.run.smithi188.stdout:(31/136): ceph-mgr-dashboard-19.3.0-4872.g0d613 22 MB/s | 14 MB 00:00 2024-09-11T14:34:16.854 INFO:teuthology.orchestra.run.smithi188.stdout:(32/136): ceph-mgr-rook-19.3.0-4872.g0d61375b.e 295 kB/s | 49 kB 00:00 2024-09-11T14:34:16.879 INFO:teuthology.orchestra.run.smithi188.stdout:(33/136): ceph-prometheus-alerts-19.3.0-4872.g0 327 kB/s | 16 kB 00:00 2024-09-11T14:34:16.904 INFO:teuthology.orchestra.run.smithi188.stdout:(34/136): ceph-volume-19.3.0-4872.g0d61375b.el9 5.5 MB/s | 285 kB 00:00 2024-09-11T14:34:16.946 INFO:teuthology.orchestra.run.smithi188.stdout:(35/136): cephadm-19.3.0-4872.g0d61375b.el9.noa 11 MB/s | 784 kB 00:00 2024-09-11T14:34:18.555 INFO:teuthology.orchestra.run.smithi188.stdout:(36/136): fuse-2.9.9-16.el9.x86_64.rpm 48 kB/s | 80 kB 00:01 2024-09-11T14:34:18.747 INFO:teuthology.orchestra.run.smithi188.stdout:(37/136): libgfortran-11.5.0-2.el9.x86_64.rpm 445 kB/s | 801 kB 00:01 2024-09-11T14:34:19.430 INFO:teuthology.orchestra.run.smithi188.stdout:(38/136): libquadmath-11.5.0-2.el9.x86_64.rpm 218 kB/s | 191 kB 00:00 2024-09-11T14:34:19.497 INFO:teuthology.orchestra.run.smithi188.stdout:(39/136): lksctp-tools-1.0.19-2.el9.x86_64.rpm 125 kB/s | 94 kB 00:00 2024-09-11T14:34:20.847 INFO:teuthology.orchestra.run.smithi188.stdout:(40/136): ceph-test-19.3.0-4872.g0d61375b.el9.x 12 MB/s | 85 MB 00:07 2024-09-11T14:34:20.872 INFO:teuthology.orchestra.run.smithi188.stdout:(41/136): python3-cffi-1.14.5-5.el9.x86_64.rpm 176 kB/s | 253 kB 00:01 2024-09-11T14:34:21.223 INFO:teuthology.orchestra.run.smithi188.stdout:(42/136): python3-cryptography-36.0.1-4.el9.x86 728 kB/s | 1.2 MB 00:01 2024-09-11T14:34:21.581 INFO:teuthology.orchestra.run.smithi188.stdout:(43/136): python3-ply-3.11-14.el9.noarch.rpm 145 kB/s | 106 kB 00:00 2024-09-11T14:34:21.682 INFO:teuthology.orchestra.run.smithi188.stdout:(44/136): python3-pycparser-2.20-6.el9.noarch.r 167 kB/s | 135 kB 00:00 2024-09-11T14:34:21.824 INFO:teuthology.orchestra.run.smithi188.stdout:(45/136): boost-program-options-1.75.0-8.el9.x8 757 kB/s | 107 kB 00:00 2024-09-11T14:34:21.874 INFO:teuthology.orchestra.run.smithi188.stdout:(46/136): flexiblas-3.0.4-8.el9.x86_64.rpm 652 kB/s | 33 kB 00:00 2024-09-11T14:34:22.016 INFO:teuthology.orchestra.run.smithi188.stdout:(47/136): python3-requests-2.25.1-8.el9.noarch. 158 kB/s | 125 kB 00:00 2024-09-11T14:34:22.108 INFO:teuthology.orchestra.run.smithi188.stdout:(48/136): flexiblas-openblas-openmp-3.0.4-8.el9 199 kB/s | 18 kB 00:00 2024-09-11T14:34:22.300 INFO:teuthology.orchestra.run.smithi188.stdout:(49/136): libasan8-14.2.1-1.2.el9.x86_64.rpm 2.6 MB/s | 513 kB 00:00 2024-09-11T14:34:22.392 INFO:teuthology.orchestra.run.smithi188.stdout:(50/136): libnbd-1.20.2-2.el9.x86_64.rpm 1.8 MB/s | 171 kB 00:00 2024-09-11T14:34:22.525 INFO:teuthology.orchestra.run.smithi188.stdout:(51/136): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-09-11T14:34:22.559 INFO:teuthology.orchestra.run.smithi188.stdout:(52/136): python3-urllib3-1.26.5-6.el9.noarch.r 220 kB/s | 215 kB 00:00 2024-09-11T14:34:22.584 INFO:teuthology.orchestra.run.smithi188.stdout:(53/136): librabbitmq-0.11.0-7.el9.x86_64.rpm 775 kB/s | 45 kB 00:00 2024-09-11T14:34:22.668 INFO:teuthology.orchestra.run.smithi188.stdout:(54/136): libubsan-11.5.0-2.el9.x86_64.rpm 2.1 MB/s | 183 kB 00:00 2024-09-11T14:34:22.727 INFO:teuthology.orchestra.run.smithi188.stdout:(55/136): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.5 MB/s | 3.0 MB 00:00 2024-09-11T14:34:22.818 INFO:teuthology.orchestra.run.smithi188.stdout:(56/136): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 292 kB 00:00 2024-09-11T14:34:22.852 INFO:teuthology.orchestra.run.smithi188.stdout:(57/136): librdkafka-1.6.1-102.el9.x86_64.rpm 2.2 MB/s | 662 kB 00:00 2024-09-11T14:34:22.877 INFO:teuthology.orchestra.run.smithi188.stdout:(58/136): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-09-11T14:34:22.903 INFO:teuthology.orchestra.run.smithi188.stdout:(59/136): openblas-0.3.26-2.el9.x86_64.rpm 458 kB/s | 38 kB 00:00 2024-09-11T14:34:22.953 INFO:teuthology.orchestra.run.smithi188.stdout:(60/136): python3-attrs-20.3.0-7.el9.noarch.rpm 1.7 MB/s | 87 kB 00:00 2024-09-11T14:34:23.195 INFO:teuthology.orchestra.run.smithi188.stdout:(61/136): protobuf-3.14.0-13.el9.x86_64.rpm 3.2 MB/s | 1.0 MB 00:00 2024-09-11T14:34:23.304 INFO:teuthology.orchestra.run.smithi188.stdout:(62/136): python3-devel-3.9.19-8.el9.x86_64.rpm 2.2 MB/s | 245 kB 00:00 2024-09-11T14:34:23.354 INFO:teuthology.orchestra.run.smithi188.stdout:(63/136): python3-iniconfig-1.1.1-7.el9.noarch. 350 kB/s | 17 kB 00:00 2024-09-11T14:34:23.471 INFO:teuthology.orchestra.run.smithi188.stdout:(64/136): python3-jinja2-2.11.3-6.el9.noarch.rp 2.1 MB/s | 248 kB 00:00 2024-09-11T14:34:23.513 INFO:teuthology.orchestra.run.smithi188.stdout:(65/136): python3-jsonpatch-1.21-16.el9.noarch. 614 kB/s | 26 kB 00:00 2024-09-11T14:34:23.556 INFO:teuthology.orchestra.run.smithi188.stdout:(66/136): python3-jsonpointer-2.0-4.el9.noarch. 461 kB/s | 19 kB 00:00 2024-09-11T14:34:23.630 INFO:teuthology.orchestra.run.smithi188.stdout:(67/136): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-09-11T14:34:23.739 INFO:teuthology.orchestra.run.smithi188.stdout:(68/136): python3-markupsafe-1.1.1-12.el9.x86_6 320 kB/s | 35 kB 00:00 2024-09-11T14:34:24.114 INFO:teuthology.orchestra.run.smithi188.stdout:(69/136): openblas-openmp-0.3.26-2.el9.x86_64.r 3.9 MB/s | 4.9 MB 00:01 2024-09-11T14:34:24.206 INFO:teuthology.orchestra.run.smithi188.stdout:(70/136): python3-babel-2.9.1-2.el9.noarch.rpm 4.8 MB/s | 6.0 MB 00:01 2024-09-11T14:34:24.248 INFO:teuthology.orchestra.run.smithi188.stdout:(71/136): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.8 MB/s | 250 kB 00:00 2024-09-11T14:34:24.298 INFO:teuthology.orchestra.run.smithi188.stdout:(72/136): python3-oauthlib-3.1.1-5.el9.noarch.r 2.4 MB/s | 222 kB 00:00 2024-09-11T14:34:24.323 INFO:teuthology.orchestra.run.smithi188.stdout:(73/136): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 77 kB 00:00 2024-09-11T14:34:24.349 INFO:teuthology.orchestra.run.smithi188.stdout:(74/136): python3-pluggy-0.13.1-7.el9.noarch.rp 824 kB/s | 41 kB 00:00 2024-09-11T14:34:24.376 INFO:teuthology.orchestra.run.smithi188.stdout:(75/136): python3-prettytable-0.7.2-27.el9.noar 840 kB/s | 42 kB 00:00 2024-09-11T14:34:24.433 INFO:teuthology.orchestra.run.smithi188.stdout:(76/136): python3-protobuf-3.14.0-13.el9.noarch 3.2 MB/s | 269 kB 00:00 2024-09-11T14:34:24.491 INFO:teuthology.orchestra.run.smithi188.stdout:(77/136): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-09-11T14:34:24.525 INFO:teuthology.orchestra.run.smithi188.stdout:(78/136): python3-py-1.10.0-6.el9.noarch.rpm 3.1 MB/s | 477 kB 00:00 2024-09-11T14:34:24.575 INFO:teuthology.orchestra.run.smithi188.stdout:(79/136): python3-pyasn1-modules-0.4.8-6.el9.no 3.3 MB/s | 279 kB 00:00 2024-09-11T14:34:24.625 INFO:teuthology.orchestra.run.smithi188.stdout:(80/136): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-11T14:34:24.676 INFO:teuthology.orchestra.run.smithi188.stdout:(81/136): python3-requests-oauthlib-1.3.0-12.el 1.1 MB/s | 54 kB 00:00 2024-09-11T14:34:24.709 INFO:teuthology.orchestra.run.smithi188.stdout:(82/136): python3-pytest-6.2.2-7.el9.noarch.rpm 2.8 MB/s | 519 kB 00:00 2024-09-11T14:34:24.768 INFO:teuthology.orchestra.run.smithi188.stdout:(83/136): python3-toml-0.10.2-6.el9.noarch.rpm 717 kB/s | 42 kB 00:00 2024-09-11T14:34:24.860 INFO:teuthology.orchestra.run.smithi188.stdout:(84/136): qatlib-24.02.0-1.el9.x86_64.rpm 2.4 MB/s | 221 kB 00:00 2024-09-11T14:34:24.910 INFO:teuthology.orchestra.run.smithi188.stdout:(85/136): qatlib-service-24.02.0-1.el9.x86_64.r 718 kB/s | 36 kB 00:00 2024-09-11T14:34:24.961 INFO:teuthology.orchestra.run.smithi188.stdout:(86/136): qatzip-libs-1.2.0-1.el9.x86_64.rpm 948 kB/s | 47 kB 00:00 2024-09-11T14:34:25.044 INFO:teuthology.orchestra.run.smithi188.stdout:(87/136): socat-1.7.4.1-6.el9.x86_64.rpm 3.6 MB/s | 304 kB 00:00 2024-09-11T14:34:25.103 INFO:teuthology.orchestra.run.smithi188.stdout:(88/136): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-09-11T14:34:25.387 INFO:teuthology.orchestra.run.smithi188.stdout:(89/136): lua-devel-5.4.4-4.el9.x86_64.rpm 79 kB/s | 22 kB 00:00 2024-09-11T14:34:25.462 INFO:teuthology.orchestra.run.smithi188.stdout:(90/136): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.0 MB/s | 5.1 MB 00:01 2024-09-11T14:34:25.512 INFO:teuthology.orchestra.run.smithi188.stdout:(91/136): abseil-cpp-20211102.0-3.el9.x86_64.rp 11 MB/s | 548 kB 00:00 2024-09-11T14:34:25.546 INFO:teuthology.orchestra.run.smithi188.stdout:(92/136): gperftools-libs-2.9.1-2.el9.x86_64.rp 9.1 MB/s | 309 kB 00:00 2024-09-11T14:34:25.571 INFO:teuthology.orchestra.run.smithi188.stdout:(93/136): grpc-data-1.46.7-10.el9.noarch.rpm 783 kB/s | 19 kB 00:00 2024-09-11T14:34:25.688 INFO:teuthology.orchestra.run.smithi188.stdout:(94/136): libarrow-9.0.0-11.el9.x86_64.rpm 38 MB/s | 4.4 MB 00:00 2024-09-11T14:34:25.713 INFO:teuthology.orchestra.run.smithi188.stdout:(95/136): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-09-11T14:34:25.783 INFO:teuthology.orchestra.run.smithi188.stdout:(96/136): liboath-2.6.7-2.el9.x86_64.rpm 696 kB/s | 49 kB 00:00 2024-09-11T14:34:25.814 INFO:teuthology.orchestra.run.smithi188.stdout:(97/136): libunwind-1.6.2-1.el9.x86_64.rpm 2.2 MB/s | 67 kB 00:00 2024-09-11T14:34:25.839 INFO:teuthology.orchestra.run.smithi188.stdout:(98/136): luarocks-3.9.2-5.el9.noarch.rpm 5.9 MB/s | 151 kB 00:00 2024-09-11T14:34:25.881 INFO:teuthology.orchestra.run.smithi188.stdout:(99/136): parquet-libs-9.0.0-11.el9.x86_64.rpm 20 MB/s | 839 kB 00:00 2024-09-11T14:34:25.915 INFO:teuthology.orchestra.run.smithi188.stdout:(100/136): python3-asyncssh-2.13.2-5.el9.noarch 16 MB/s | 548 kB 00:00 2024-09-11T14:34:25.932 INFO:teuthology.orchestra.run.smithi188.stdout:(101/136): python3-bcrypt-3.2.2-1.el9.x86_64.rp 2.5 MB/s | 43 kB 00:00 2024-09-11T14:34:25.949 INFO:teuthology.orchestra.run.smithi188.stdout:(102/136): python3-cachetools-4.2.4-1.el9.noarc 1.9 MB/s | 32 kB 00:00 2024-09-11T14:34:25.966 INFO:teuthology.orchestra.run.smithi188.stdout:(103/136): python3-certifi-2023.05.07-4.el9.noa 849 kB/s | 14 kB 00:00 2024-09-11T14:34:25.991 INFO:teuthology.orchestra.run.smithi188.stdout:(104/136): python3-cheroot-8.6.0-4.el9.noarch.r 6.7 MB/s | 172 kB 00:00 2024-09-11T14:34:26.033 INFO:teuthology.orchestra.run.smithi188.stdout:(105/136): python3-cherrypy-18.6.1-2.el9.noarch 8.4 MB/s | 358 kB 00:00 2024-09-11T14:34:26.058 INFO:teuthology.orchestra.run.smithi188.stdout:(106/136): python3-google-auth-2.34.0-1.el9.noa 8.6 MB/s | 220 kB 00:00 2024-09-11T14:34:26.084 INFO:teuthology.orchestra.run.smithi188.stdout:(107/136): protobuf-compiler-3.14.0-13.el9.x86_ 1.2 MB/s | 863 kB 00:00 2024-09-11T14:34:26.134 INFO:teuthology.orchestra.run.smithi188.stdout:(108/136): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-09-11T14:34:26.159 INFO:teuthology.orchestra.run.smithi188.stdout:(109/136): python3-jaraco-8.2.1-3.el9.noarch.rp 429 kB/s | 11 kB 00:00 2024-09-11T14:34:26.201 INFO:teuthology.orchestra.run.smithi188.stdout:(110/136): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-09-11T14:34:26.227 INFO:teuthology.orchestra.run.smithi188.stdout:(111/136): python3-jaraco-classes-3.2.1-5.el9.n 264 kB/s | 18 kB 00:00 2024-09-11T14:34:26.299 INFO:teuthology.orchestra.run.smithi188.stdout:(112/136): python3-jaraco-collections-3.0.0-8.e 238 kB/s | 23 kB 00:00 2024-09-11T14:34:26.328 INFO:teuthology.orchestra.run.smithi188.stdout:(113/136): python3-jaraco-functools-3.5.0-2.el9 193 kB/s | 19 kB 00:00 2024-09-11T14:34:26.353 INFO:teuthology.orchestra.run.smithi188.stdout:(114/136): python3-jaraco-text-3.2.0-6.el9.noar 368 kB/s | 20 kB 00:00 2024-09-11T14:34:26.378 INFO:teuthology.orchestra.run.smithi188.stdout:(115/136): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-11T14:34:26.412 INFO:teuthology.orchestra.run.smithi188.stdout:(116/136): python3-more-itertools-8.12.0-2.el9. 2.3 MB/s | 79 kB 00:00 2024-09-11T14:34:26.437 INFO:teuthology.orchestra.run.smithi188.stdout:(117/136): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-09-11T14:34:26.471 INFO:teuthology.orchestra.run.smithi188.stdout:(118/136): python3-kubernetes-26.1.0-2.el9.noar 7.2 MB/s | 1.0 MB 00:00 2024-09-11T14:34:26.496 INFO:teuthology.orchestra.run.smithi188.stdout:(119/136): python3-portend-3.1.0-2.el9.noarch.r 663 kB/s | 16 kB 00:00 2024-09-11T14:34:26.521 INFO:teuthology.orchestra.run.smithi188.stdout:(120/136): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-09-11T14:34:26.547 INFO:teuthology.orchestra.run.smithi188.stdout:(121/136): python3-repoze-lru-0.7-16.el9.noarch 1.2 MB/s | 31 kB 00:00 2024-09-11T14:34:26.572 INFO:teuthology.orchestra.run.smithi188.stdout:(122/136): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-09-11T14:34:26.597 INFO:teuthology.orchestra.run.smithi188.stdout:(123/136): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-11T14:34:26.623 INFO:teuthology.orchestra.run.smithi188.stdout:(124/136): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-09-11T14:34:26.648 INFO:teuthology.orchestra.run.smithi188.stdout:(125/136): python3-tempora-5.0.0-2.el9.noarch.r 713 kB/s | 36 kB 00:00 2024-09-11T14:34:26.673 INFO:teuthology.orchestra.run.smithi188.stdout:(126/136): python3-typing-extensions-4.12.2-2.e 1.4 MB/s | 74 kB 00:00 2024-09-11T14:34:26.699 INFO:teuthology.orchestra.run.smithi188.stdout:(127/136): python3-websocket-client-1.2.3-2.el9 3.5 MB/s | 90 kB 00:00 2024-09-11T14:34:26.732 INFO:teuthology.orchestra.run.smithi188.stdout:(128/136): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-09-11T14:34:26.757 INFO:teuthology.orchestra.run.smithi188.stdout:(129/136): python3-zc-lockfile-2.0-10.el9.noarc 807 kB/s | 20 kB 00:00 2024-09-11T14:34:26.783 INFO:teuthology.orchestra.run.smithi188.stdout:(130/136): re2-20211101-20.el9.x86_64.rpm 7.5 MB/s | 191 kB 00:00 2024-09-11T14:34:26.808 INFO:teuthology.orchestra.run.smithi188.stdout:(131/136): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-09-11T14:34:26.833 INFO:teuthology.orchestra.run.smithi188.stdout:(132/136): yaml-cpp-0.6.3-5.el9.x86_64.rpm 4.7 MB/s | 122 kB 00:00 2024-09-11T14:34:26.884 INFO:teuthology.orchestra.run.smithi188.stdout:(133/136): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-11T14:34:27.151 INFO:teuthology.orchestra.run.smithi188.stdout:(134/136): librbd1-19.3.0-4872.g0d61375b.el9.x8 14 MB/s | 3.8 MB 00:00 2024-09-11T14:34:27.209 INFO:teuthology.orchestra.run.smithi188.stdout:(135/136): librados2-19.3.0-4872.g0d61375b.el9. 9.2 MB/s | 3.5 MB 00:00 2024-09-11T14:34:27.368 INFO:teuthology.orchestra.run.smithi188.stdout:(136/136): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.0 MB/s | 16 MB 00:02 2024-09-11T14:34:27.373 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:34:27.374 INFO:teuthology.orchestra.run.smithi188.stdout:Total 19 MB/s | 305 MB 00:15 2024-09-11T14:34:28.284 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-09-11T14:34:28.391 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-09-11T14:34:28.391 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-09-11T14:34:31.002 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-09-11T14:34:31.003 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-09-11T14:34:36.991 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-09-11T14:34:37.061 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/138 2024-09-11T14:34:37.179 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/138 2024-09-11T14:34:37.347 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : liboath-2.6.7-2.el9.x86_64 3/138 2024-09-11T14:34:37.454 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/138 2024-09-11T14:34:37.910 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 5/138 2024-09-11T14:34:38.005 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librados2-2:19.3.0-4872.g0d61375b.el9.x86_64 6/138 2024-09-11T14:34:38.129 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-2:19.3.0-4872.g0d61375b.el9.x86_64 6/138 2024-09-11T14:34:38.156 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs2-2:19.3.0-4872.g0d61375b.el9.x86_64 7/138 2024-09-11T14:34:38.231 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephfs2-2:19.3.0-4872.g0d61375b.el9.x86_64 7/138 2024-09-11T14:34:38.281 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rados-2:19.3.0-4872.g0d61375b.el9.x86_64 8/138 2024-09-11T14:34:38.500 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 9/138 2024-09-11T14:34:38.500 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 10/138 2024-09-11T14:34:38.500 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 11/138 2024-09-11T14:34:38.500 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-packaging-20.9-5.el9.noarch 12/138 2024-09-11T14:34:38.780 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 13/138 2024-09-11T14:34:38.836 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librbd1-2:19.3.0-4872.g0d61375b.el9.x86_64 14/138 2024-09-11T14:34:38.912 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-2:19.3.0-4872.g0d61375b.el9.x86_64 14/138 2024-09-11T14:34:38.937 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephsqlite-2:19.3.0-4872.g0d61375b.el9.x86_64 15/138 2024-09-11T14:34:39.055 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4872.g0d61375b.el9.x86_64 15/138 2024-09-11T14:34:39.091 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libradosstriper1-2:19.3.0-4872.g0d61375b.el9.x86 16/138 2024-09-11T14:34:39.154 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4872.g0d61375b.el9.x86 16/138 2024-09-11T14:34:39.176 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 17/138 2024-09-11T14:34:39.198 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 17/138 2024-09-11T14:34:39.345 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 18/138 2024-09-11T14:34:39.375 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 18/138 2024-09-11T14:34:39.409 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 19/138 2024-09-11T14:34:39.517 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : re2-1:20211101-20.el9.x86_64 20/138 2024-09-11T14:34:39.556 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/138 2024-09-11T14:34:39.586 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 22/138 2024-09-11T14:34:39.627 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 23/138 2024-09-11T14:34:39.693 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/138 2024-09-11T14:34:39.743 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/138 2024-09-11T14:34:39.777 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/138 2024-09-11T14:34:39.806 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/138 2024-09-11T14:34:39.845 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libubsan-11.5.0-2.el9.x86_64 28/138 2024-09-11T14:34:39.869 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libasan8-14.2.1-1.2.el9.x86_64 29/138 2024-09-11T14:34:39.933 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libasan8-14.2.1-1.2.el9.x86_64 29/138 2024-09-11T14:34:40.002 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/138 2024-09-11T14:34:40.072 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 31/138 2024-09-11T14:34:40.116 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/138 2024-09-11T14:34:40.149 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 33/138 2024-09-11T14:34:40.189 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 34/138 2024-09-11T14:34:40.266 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 35/138 2024-09-11T14:34:40.299 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-common-2:19.3.0-4872.g0d61375b.el9. 36/138 2024-09-11T14:34:40.342 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-argparse-2:19.3.0-4872.g0d61375b.el 37/138 2024-09-11T14:34:40.371 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cephfs-2:19.3.0-4872.g0d61375b.el9.x86_6 38/138 2024-09-11T14:34:40.442 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 39/138 2024-09-11T14:34:40.519 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : luarocks-3.9.2-5.el9.noarch 40/138 2024-09-11T14:34:40.663 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/138 2024-09-11T14:34:40.713 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/138 2024-09-11T14:34:41.240 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/138 2024-09-11T14:34:41.325 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/138 2024-09-11T14:34:41.371 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/138 2024-09-11T14:34:41.406 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 46/138 2024-09-11T14:34:41.440 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 47/138 2024-09-11T14:34:41.472 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 48/138 2024-09-11T14:34:41.531 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 49/138 2024-09-11T14:34:41.566 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rbd-2:19.3.0-4872.g0d61375b.el9.x86_64 50/138 2024-09-11T14:34:41.611 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librados-devel-2:19.3.0-4872.g0d61375b.el9.x86_6 51/138 2024-09-11T14:34:41.647 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 52/138 2024-09-11T14:34:41.685 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/138 2024-09-11T14:34:41.749 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 54/138 2024-09-11T14:34:41.781 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : thrift-0.15.0-2.el9.x86_64 55/138 2024-09-11T14:34:41.872 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 56/138 2024-09-11T14:34:41.936 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 57/138 2024-09-11T14:34:42.010 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 58/138 2024-09-11T14:34:42.041 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 59/138 2024-09-11T14:34:42.086 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 60/138 2024-09-11T14:34:42.141 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 61/138 2024-09-11T14:34:42.187 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 62/138 2024-09-11T14:34:42.228 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 63/138 2024-09-11T14:34:42.365 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 64/138 2024-09-11T14:34:42.405 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 65/138 2024-09-11T14:34:42.447 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 66/138 2024-09-11T14:34:42.549 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 67/138 2024-09-11T14:34:42.589 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 68/138 2024-09-11T14:34:42.696 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 69/138 2024-09-11T14:34:42.796 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 70/138 2024-09-11T14:34:42.821 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 71/138 2024-09-11T14:34:42.931 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 72/138 2024-09-11T14:34:43.056 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 73/138 2024-09-11T14:34:43.090 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 74/138 2024-09-11T14:34:43.137 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 75/138 2024-09-11T14:34:43.443 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 76/138 2024-09-11T14:34:43.544 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-py-1.10.0-6.el9.noarch 77/138 2024-09-11T14:34:43.587 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 78/138 2024-09-11T14:34:43.619 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 79/138 2024-09-11T14:34:43.658 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 80/138 2024-09-11T14:34:43.773 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pluggy-0.13.1-7.el9.noarch 81/138 2024-09-11T14:34:43.829 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/138 2024-09-11T14:34:44.548 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/138 2024-09-11T14:34:44.601 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/138 2024-09-11T14:34:44.643 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/138 2024-09-11T14:34:44.677 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/138 2024-09-11T14:34:44.793 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-iniconfig-1.1.1-7.el9.noarch 87/138 2024-09-11T14:34:44.852 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 88/138 2024-09-11T14:34:44.881 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/138 2024-09-11T14:34:45.065 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-0.3.26-2.el9.x86_64 90/138 2024-09-11T14:34:45.087 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 91/138 2024-09-11T14:34:45.174 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 92/138 2024-09-11T14:34:45.829 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 93/138 2024-09-11T14:34:45.955 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 94/138 2024-09-11T14:34:47.213 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 95/138 2024-09-11T14:34:47.288 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 96/138 2024-09-11T14:34:47.331 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 97/138 2024-09-11T14:34:47.922 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 98/138 2024-09-11T14:34:47.951 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librgw2-2:19.3.0-4872.g0d61375b.el9.x86_64 99/138 2024-09-11T14:34:48.015 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librgw2-2:19.3.0-4872.g0d61375b.el9.x86_64 99/138 2024-09-11T14:34:48.045 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rgw-2:19.3.0-4872.g0d61375b.el9.x86_64 100/138 2024-09-11T14:34:50.535 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 101/138 2024-09-11T14:34:50.567 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 101/138 2024-09-11T14:34:51.265 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 101/138 2024-09-11T14:34:51.306 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-base-2:19.3.0-4872.g0d61375b.el9.x86_64 102/138 2024-09-11T14:34:51.389 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-base-2:19.3.0-4872.g0d61375b.el9.x86_64 102/138 2024-09-11T14:34:51.389 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-11T14:34:51.389 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-11T14:34:51.389 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:34:51.432 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-selinux-2:19.3.0-4872.g0d61375b.el9.x86_64 103/138 2024-09-11T14:35:03.648 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4872.g0d61375b.el9.x86_64 103/138 2024-09-11T14:35:03.648 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /sys 2024-09-11T14:35:03.648 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /proc 2024-09-11T14:35:03.648 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /mnt 2024-09-11T14:35:03.649 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /var/tmp 2024-09-11T14:35:03.649 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /home 2024-09-11T14:35:03.649 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /root 2024-09-11T14:35:03.649 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /tmp 2024-09-11T14:35:03.649 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:03.818 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-volume-2:19.3.0-4872.g0d61375b.el9.noarch 104/138 2024-09-11T14:35:03.834 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-volume-2:19.3.0-4872.g0d61375b.el9.noarch 104/138 2024-09-11T14:35:03.834 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:35:03.834 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-11T14:35:03.834 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:14.404 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-osd-2:19.3.0-4872.g0d61375b.el9.x86_64 105/138 2024-09-11T14:35:14.431 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-osd-2:19.3.0-4872.g0d61375b.el9.x86_64 105/138 2024-09-11T14:35:14.431 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:35:14.432 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-11T14:35:14.432 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T14:35:14.432 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T14:35:14.432 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:14.672 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mds-2:19.3.0-4872.g0d61375b.el9.x86_64 106/138 2024-09-11T14:35:14.696 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mds-2:19.3.0-4872.g0d61375b.el9.x86_64 106/138 2024-09-11T14:35:14.696 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:35:14.696 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-11T14:35:14.696 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T14:35:14.696 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T14:35:14.696 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:15.247 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mon-2:19.3.0-4872.g0d61375b.el9.x86_64 107/138 2024-09-11T14:35:15.271 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mon-2:19.3.0-4872.g0d61375b.el9.x86_64 107/138 2024-09-11T14:35:15.271 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:35:15.271 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-11T14:35:15.271 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T14:35:15.271 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T14:35:15.271 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:15.334 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ply-3.11-14.el9.noarch 108/138 2024-09-11T14:35:15.388 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 109/138 2024-09-11T14:35:15.571 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 110/138 2024-09-11T14:35:15.622 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 111/138 2024-09-11T14:35:15.690 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 112/138 2024-09-11T14:35:15.778 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 113/138 2024-09-11T14:35:15.914 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 114/138 2024-09-11T14:35:15.974 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 115/138 2024-09-11T14:35:16.012 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 116/138 2024-09-11T14:35:16.040 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : fuse-2.9.9-16.el9.x86_64 117/138 2024-09-11T14:35:16.530 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 118/138 2024-09-11T14:35:16.556 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 118/138 2024-09-11T14:35:16.588 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4872.g0d61375b.e 119/138 2024-09-11T14:35:16.663 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4872.g0d61375b. 120/138 2024-09-11T14:35:16.696 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4872.g0d61375b.el9.noa 121/138 2024-09-11T14:35:19.853 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4872.g0d61375b.el9.noa 121/138 2024-09-11T14:35:19.985 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4872.g0d61375b.el9.n 122/138 2024-09-11T14:35:20.948 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4872.g0d61375b.el9.n 122/138 2024-09-11T14:35:20.977 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4872.g0d6 123/138 2024-09-11T14:35:21.090 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4872.g0d6 123/138 2024-09-11T14:35:21.280 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4872.g0d61375b.el 124/138 2024-09-11T14:35:21.305 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-2:19.3.0-4872.g0d61375b.el9.x86_64 125/138 2024-09-11T14:35:21.329 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4872.g0d61375b.el9.x86_64 125/138 2024-09-11T14:35:21.329 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:35:21.329 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-11T14:35:21.329 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T14:35:21.329 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T14:35:21.329 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:21.380 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-rook-2:19.3.0-4872.g0d61375b.el9.noarch 126/138 2024-09-11T14:35:21.392 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4872.g0d61375b.el9.noarch 126/138 2024-09-11T14:35:21.507 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-2:19.3.0-4872.g0d61375b.el9.x86_64 127/138 2024-09-11T14:35:22.557 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 128/138 2024-09-11T14:35:22.590 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-radosgw-2:19.3.0-4872.g0d61375b.el9.x86_64 129/138 2024-09-11T14:35:22.614 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4872.g0d61375b.el9.x86_64 129/138 2024-09-11T14:35:22.614 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:35:22.614 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-11T14:35:22.614 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T14:35:22.614 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T14:35:22.614 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:22.658 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4872.g0d613 130/138 2024-09-11T14:35:22.681 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4872.g0d613 130/138 2024-09-11T14:35:22.681 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:35:22.681 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-11T14:35:22.681 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:23.133 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-mirror-2:19.3.0-4872.g0d61375b.el9.x86_64 131/138 2024-09-11T14:35:23.156 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4872.g0d61375b.el9.x86_64 131/138 2024-09-11T14:35:23.157 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:35:23.157 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-11T14:35:23.157 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T14:35:23.157 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T14:35:23.157 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:34.396 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-test-2:19.3.0-4872.g0d61375b.el9.x86_64 132/138 2024-09-11T14:35:34.442 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pytest-6.2.2-7.el9.noarch 133/138 2024-09-11T14:35:34.488 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs-devel-2:19.3.0-4872.g0d61375b.el9.x86_ 134/138 2024-09-11T14:35:34.530 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 135/138 2024-09-11T14:35:34.557 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-nbd-2:19.3.0-4872.g0d61375b.el9.x86_64 136/138 2024-09-11T14:35:34.557 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 137/138 2024-09-11T14:35:34.650 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 137/138 2024-09-11T14:35:34.650 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 138/138 2024-09-11T14:35:49.713 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 138/138 2024-09-11T14:35:49.713 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-2:19.3.0-4872.g0d61375b.el9.x86_64 1/138 2024-09-11T14:35:49.713 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-base-2:19.3.0-4872.g0d61375b.el9.x86_64 2/138 2024-09-11T14:35:49.713 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 3/138 2024-09-11T14:35:49.713 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 4/138 2024-09-11T14:35:49.713 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4872.g0d613 5/138 2024-09-11T14:35:49.714 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mds-2:19.3.0-4872.g0d61375b.el9.x86_64 6/138 2024-09-11T14:35:49.714 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-2:19.3.0-4872.g0d61375b.el9.x86_64 7/138 2024-09-11T14:35:49.714 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mon-2:19.3.0-4872.g0d61375b.el9.x86_64 8/138 2024-09-11T14:35:49.714 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-osd-2:19.3.0-4872.g0d61375b.el9.x86_64 9/138 2024-09-11T14:35:49.714 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-radosgw-2:19.3.0-4872.g0d61375b.el9.x86_64 10/138 2024-09-11T14:35:49.714 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-selinux-2:19.3.0-4872.g0d61375b.el9.x86_64 11/138 2024-09-11T14:35:49.714 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-test-2:19.3.0-4872.g0d61375b.el9.x86_64 12/138 2024-09-11T14:35:49.714 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs-devel-2:19.3.0-4872.g0d61375b.el9.x86_ 13/138 2024-09-11T14:35:49.714 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs2-2:19.3.0-4872.g0d61375b.el9.x86_64 14/138 2024-09-11T14:35:49.714 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephsqlite-2:19.3.0-4872.g0d61375b.el9.x86_64 15/138 2024-09-11T14:35:49.714 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados-devel-2:19.3.0-4872.g0d61375b.el9.x86_6 16/138 2024-09-11T14:35:49.714 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libradosstriper1-2:19.3.0-4872.g0d61375b.el9.x86 17/138 2024-09-11T14:35:49.714 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librgw2-2:19.3.0-4872.g0d61375b.el9.x86_64 18/138 2024-09-11T14:35:49.715 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4872.g0d61375b.el 19/138 2024-09-11T14:35:49.715 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-common-2:19.3.0-4872.g0d61375b.el9. 20/138 2024-09-11T14:35:49.715 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cephfs-2:19.3.0-4872.g0d61375b.el9.x86_6 21/138 2024-09-11T14:35:49.715 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rados-2:19.3.0-4872.g0d61375b.el9.x86_64 22/138 2024-09-11T14:35:49.715 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rbd-2:19.3.0-4872.g0d61375b.el9.x86_64 23/138 2024-09-11T14:35:49.715 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rgw-2:19.3.0-4872.g0d61375b.el9.x86_64 24/138 2024-09-11T14:35:49.715 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 25/138 2024-09-11T14:35:49.715 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-mirror-2:19.3.0-4872.g0d61375b.el9.x86_64 26/138 2024-09-11T14:35:49.715 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-nbd-2:19.3.0-4872.g0d61375b.el9.x86_64 27/138 2024-09-11T14:35:49.715 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4872.g0d61375b. 28/138 2024-09-11T14:35:49.715 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4872.g0d61375b.el9.noa 29/138 2024-09-11T14:35:49.728 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4872.g0d61375b.el9.n 30/138 2024-09-11T14:35:49.728 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4872.g0d6 31/138 2024-09-11T14:35:49.728 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4872.g0d61375b.el 32/138 2024-09-11T14:35:49.728 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4872.g0d61375b.el9.noarch 33/138 2024-09-11T14:35:49.728 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4872.g0d61375b.e 34/138 2024-09-11T14:35:49.728 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-volume-2:19.3.0-4872.g0d61375b.el9.noarch 35/138 2024-09-11T14:35:49.728 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 36/138 2024-09-11T14:35:49.729 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 37/138 2024-09-11T14:35:49.729 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 38/138 2024-09-11T14:35:49.729 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 39/138 2024-09-11T14:35:49.729 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 40/138 2024-09-11T14:35:49.729 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 41/138 2024-09-11T14:35:49.729 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 42/138 2024-09-11T14:35:49.729 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ply-3.11-14.el9.noarch 43/138 2024-09-11T14:35:49.729 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 44/138 2024-09-11T14:35:49.729 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 45/138 2024-09-11T14:35:49.729 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 46/138 2024-09-11T14:35:49.729 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 47/138 2024-09-11T14:35:49.729 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 48/138 2024-09-11T14:35:49.729 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 49/138 2024-09-11T14:35:49.729 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 50/138 2024-09-11T14:35:49.729 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libasan8-14.2.1-1.2.el9.x86_64 51/138 2024-09-11T14:35:49.730 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 52/138 2024-09-11T14:35:49.730 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 53/138 2024-09-11T14:35:49.730 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 54/138 2024-09-11T14:35:49.730 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 55/138 2024-09-11T14:35:49.730 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libubsan-11.5.0-2.el9.x86_64 56/138 2024-09-11T14:35:49.730 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 57/138 2024-09-11T14:35:49.730 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lua-5.4.4-4.el9.x86_64 58/138 2024-09-11T14:35:49.730 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 59/138 2024-09-11T14:35:49.730 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 60/138 2024-09-11T14:35:49.730 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 61/138 2024-09-11T14:35:49.730 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 62/138 2024-09-11T14:35:49.730 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 63/138 2024-09-11T14:35:49.730 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 64/138 2024-09-11T14:35:49.731 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-iniconfig-1.1.1-7.el9.noarch 65/138 2024-09-11T14:35:49.731 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 66/138 2024-09-11T14:35:49.731 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 67/138 2024-09-11T14:35:49.731 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 68/138 2024-09-11T14:35:49.731 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 69/138 2024-09-11T14:35:49.731 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 70/138 2024-09-11T14:35:49.731 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 71/138 2024-09-11T14:35:49.731 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 72/138 2024-09-11T14:35:49.731 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 73/138 2024-09-11T14:35:49.731 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 74/138 2024-09-11T14:35:49.731 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pluggy-0.13.1-7.el9.noarch 75/138 2024-09-11T14:35:49.731 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/138 2024-09-11T14:35:49.731 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/138 2024-09-11T14:35:49.731 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-py-1.10.0-6.el9.noarch 78/138 2024-09-11T14:35:49.731 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 79/138 2024-09-11T14:35:49.732 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 80/138 2024-09-11T14:35:49.732 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pytest-6.2.2-7.el9.noarch 81/138 2024-09-11T14:35:49.732 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 82/138 2024-09-11T14:35:49.732 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/138 2024-09-11T14:35:49.732 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 84/138 2024-09-11T14:35:49.732 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 85/138 2024-09-11T14:35:49.732 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 86/138 2024-09-11T14:35:49.732 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 87/138 2024-09-11T14:35:49.732 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 88/138 2024-09-11T14:35:49.732 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 89/138 2024-09-11T14:35:49.732 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 90/138 2024-09-11T14:35:49.732 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 91/138 2024-09-11T14:35:49.733 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 92/138 2024-09-11T14:35:49.733 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 93/138 2024-09-11T14:35:49.733 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 94/138 2024-09-11T14:35:49.733 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 95/138 2024-09-11T14:35:49.733 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 96/138 2024-09-11T14:35:49.733 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 97/138 2024-09-11T14:35:49.733 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 98/138 2024-09-11T14:35:49.733 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 99/138 2024-09-11T14:35:49.733 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 100/138 2024-09-11T14:35:49.733 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 101/138 2024-09-11T14:35:49.734 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 102/138 2024-09-11T14:35:49.734 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 103/138 2024-09-11T14:35:49.734 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 104/138 2024-09-11T14:35:49.734 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 105/138 2024-09-11T14:35:49.734 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 106/138 2024-09-11T14:35:49.734 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 107/138 2024-09-11T14:35:49.734 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 108/138 2024-09-11T14:35:49.734 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 109/138 2024-09-11T14:35:49.734 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 110/138 2024-09-11T14:35:49.734 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 111/138 2024-09-11T14:35:49.734 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 112/138 2024-09-11T14:35:49.734 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 113/138 2024-09-11T14:35:49.734 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 114/138 2024-09-11T14:35:49.735 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 115/138 2024-09-11T14:35:49.735 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 116/138 2024-09-11T14:35:49.735 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 117/138 2024-09-11T14:35:49.735 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 118/138 2024-09-11T14:35:49.735 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 119/138 2024-09-11T14:35:49.735 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 120/138 2024-09-11T14:35:49.735 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 121/138 2024-09-11T14:35:49.735 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/138 2024-09-11T14:35:49.735 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 123/138 2024-09-11T14:35:49.735 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 124/138 2024-09-11T14:35:49.735 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 125/138 2024-09-11T14:35:49.735 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 126/138 2024-09-11T14:35:49.735 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 127/138 2024-09-11T14:35:49.736 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 128/138 2024-09-11T14:35:49.736 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 129/138 2024-09-11T14:35:49.736 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 130/138 2024-09-11T14:35:49.736 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 131/138 2024-09-11T14:35:49.736 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : re2-1:20211101-20.el9.x86_64 132/138 2024-09-11T14:35:49.736 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 133/138 2024-09-11T14:35:49.736 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 134/138 2024-09-11T14:35:49.736 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-2:19.3.0-4872.g0d61375b.el9.x86_64 135/138 2024-09-11T14:35:49.736 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 136/138 2024-09-11T14:35:49.736 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-2:19.3.0-4872.g0d61375b.el9.x86_64 137/138 2024-09-11T14:35:51.664 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 138/138 2024-09-11T14:35:51.665 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:51.665 INFO:teuthology.orchestra.run.smithi188.stdout:Upgraded: 2024-09-11T14:35:51.665 INFO:teuthology.orchestra.run.smithi188.stdout: librados2-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:35:51.665 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:35:51.665 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-09-11T14:35:51.665 INFO:teuthology.orchestra.run.smithi188.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-11T14:35:51.665 INFO:teuthology.orchestra.run.smithi188.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-11T14:35:51.665 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:35:51.665 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:35:51.665 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:35:51.665 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:35:51.665 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:35:51.665 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:35:51.666 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:35:51.666 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:35:51.666 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:35:51.666 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:35:51.666 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:35:51.666 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:35:51.666 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:35:51.666 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:35:51.666 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:35:51.666 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:35:51.666 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:35:51.666 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:35:51.667 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:35:51.667 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-volume-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:35:51.667 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:35:51.667 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-11T14:35:51.667 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-11T14:35:51.667 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-11T14:35:51.667 INFO:teuthology.orchestra.run.smithi188.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-11T14:35:51.667 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-11T14:35:51.667 INFO:teuthology.orchestra.run.smithi188.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-11T14:35:51.667 INFO:teuthology.orchestra.run.smithi188.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-11T14:35:51.667 INFO:teuthology.orchestra.run.smithi188.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-11T14:35:51.667 INFO:teuthology.orchestra.run.smithi188.stdout: libasan8-14.2.1-1.2.el9.x86_64 2024-09-11T14:35:51.667 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:35:51.667 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:35:51.668 INFO:teuthology.orchestra.run.smithi188.stdout: libcephsqlite-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:35:51.668 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-11T14:35:51.668 INFO:teuthology.orchestra.run.smithi188.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-11T14:35:51.668 INFO:teuthology.orchestra.run.smithi188.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-11T14:35:51.668 INFO:teuthology.orchestra.run.smithi188.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-11T14:35:51.668 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-11T14:35:51.668 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-11T14:35:51.668 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:35:51.668 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:35:51.668 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-11T14:35:51.668 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:35:51.668 INFO:teuthology.orchestra.run.smithi188.stdout: libubsan-11.5.0-2.el9.x86_64 2024-09-11T14:35:51.668 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-11T14:35:51.669 INFO:teuthology.orchestra.run.smithi188.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-09-11T14:35:51.669 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-11T14:35:51.669 INFO:teuthology.orchestra.run.smithi188.stdout: lua-5.4.4-4.el9.x86_64 2024-09-11T14:35:51.669 INFO:teuthology.orchestra.run.smithi188.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-11T14:35:51.669 INFO:teuthology.orchestra.run.smithi188.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-11T14:35:51.669 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-11T14:35:51.669 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-11T14:35:51.669 INFO:teuthology.orchestra.run.smithi188.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-11T14:35:51.669 INFO:teuthology.orchestra.run.smithi188.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-11T14:35:51.669 INFO:teuthology.orchestra.run.smithi188.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-11T14:35:51.669 INFO:teuthology.orchestra.run.smithi188.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-11T14:35:51.669 INFO:teuthology.orchestra.run.smithi188.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-11T14:35:51.670 INFO:teuthology.orchestra.run.smithi188.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-11T14:35:51.670 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-11T14:35:51.670 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-11T14:35:51.670 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:35:51.670 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:35:51.670 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:35:51.670 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-11T14:35:51.670 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-11T14:35:51.670 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-11T14:35:51.670 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-11T14:35:51.670 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-11T14:35:51.671 INFO:teuthology.orchestra.run.smithi188.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-11T14:35:51.671 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-11T14:35:51.671 INFO:teuthology.orchestra.run.smithi188.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-11T14:35:51.671 INFO:teuthology.orchestra.run.smithi188.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-11T14:35:51.671 INFO:teuthology.orchestra.run.smithi188.stdout: python3-iniconfig-1.1.1-7.el9.noarch 2024-09-11T14:35:51.671 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-11T14:35:51.671 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-11T14:35:51.671 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-11T14:35:51.671 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-11T14:35:51.671 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-11T14:35:51.671 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-11T14:35:51.671 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-11T14:35:51.671 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-11T14:35:51.671 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-11T14:35:51.672 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-11T14:35:51.672 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-11T14:35:51.672 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-11T14:35:51.672 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-11T14:35:51.672 INFO:teuthology.orchestra.run.smithi188.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-11T14:35:51.672 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-11T14:35:51.672 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-11T14:35:51.672 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-11T14:35:51.672 INFO:teuthology.orchestra.run.smithi188.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-11T14:35:51.672 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-11T14:35:51.672 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pluggy-0.13.1-7.el9.noarch 2024-09-11T14:35:51.672 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ply-3.11-14.el9.noarch 2024-09-11T14:35:51.672 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-11T14:35:51.673 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-11T14:35:51.673 INFO:teuthology.orchestra.run.smithi188.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-11T14:35:51.673 INFO:teuthology.orchestra.run.smithi188.stdout: python3-py-1.10.0-6.el9.noarch 2024-09-11T14:35:51.673 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-11T14:35:51.673 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-11T14:35:51.673 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-11T14:35:51.673 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-11T14:35:51.673 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytest-6.2.2-7.el9.noarch 2024-09-11T14:35:51.673 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-11T14:35:51.673 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:35:51.673 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:35:51.673 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-11T14:35:51.673 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-11T14:35:51.674 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-11T14:35:51.674 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:35:51.674 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-11T14:35:51.674 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-11T14:35:51.674 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-11T14:35:51.674 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-11T14:35:51.674 INFO:teuthology.orchestra.run.smithi188.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-11T14:35:51.674 INFO:teuthology.orchestra.run.smithi188.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-11T14:35:51.674 INFO:teuthology.orchestra.run.smithi188.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-11T14:35:51.674 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-11T14:35:51.674 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-11T14:35:51.674 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-11T14:35:51.674 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-11T14:35:51.674 INFO:teuthology.orchestra.run.smithi188.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-11T14:35:51.674 INFO:teuthology.orchestra.run.smithi188.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-11T14:35:51.675 INFO:teuthology.orchestra.run.smithi188.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-11T14:35:51.675 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:35:51.675 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:35:51.675 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:35:51.675 INFO:teuthology.orchestra.run.smithi188.stdout: re2-1:20211101-20.el9.x86_64 2024-09-11T14:35:51.675 INFO:teuthology.orchestra.run.smithi188.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-11T14:35:51.675 INFO:teuthology.orchestra.run.smithi188.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-11T14:35:51.675 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-11T14:35:51.675 INFO:teuthology.orchestra.run.smithi188.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-09-11T14:35:51.675 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:51.675 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-09-11T14:35:52.029 DEBUG:teuthology.parallel:result is None 2024-09-11T14:35:52.029 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=0d61375b8ab53be4c233f5b93e463edfdff9316a 2024-09-11T14:35:52.172 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-11T14:35:52.193 INFO:teuthology.orchestra.run.smithi188.stdout:19.3.0-4872.g0d61375b.el9 2024-09-11T14:35:52.194 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4872.g0d61375b.el9 2024-09-11T14:35:52.194 INFO:teuthology.task.install:The correct ceph version 19.3.0-4872.g0d61375b is installed. 2024-09-11T14:35:52.196 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-11T14:35:52.196 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-11T14:35:52.197 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T14:35:52.261 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-11T14:35:52.261 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-11T14:35:52.261 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T14:35:52.326 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T14:35:52.391 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-11T14:35:52.391 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-11T14:35:52.391 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T14:35:52.456 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T14:35:52.520 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-11T14:35:52.521 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-11T14:35:52.521 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T14:35:52.584 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T14:35:52.649 INFO:teuthology.run_tasks:Running task ceph... 2024-09-11T14:35:52.733 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-11T14:35:52.734 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 777 /var/log/ceph 2024-09-11T14:35:52.758 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-11T14:35:52.759 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-11T14:35:52.824 INFO:tasks.ceph:Creating extra log directories... 2024-09-11T14:35:52.824 DEBUG:teuthology.orchestra.run.smithi188:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-11T14:35:52.895 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-11T14:35:52.895 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-11T14:35:52.896 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901368', 'branch': 'main', 'description': 'crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_python_api_tests_old_format}', 'email': None, 'first_in_suite': False, 'job_id': '7901368', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'extra_system_packages': ['python3-pytest'], 'flavor': 'crimson', 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a'}}, 'workunit': {'branch': 'wip-matanb-crimson-seastore-logs', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'dbebf1a3db6db7b5579464ee715edcf1b3e9cce2'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 5813, 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-matanb-crimson-seastore-logs', 'suite_path': '/home/teuthworker/src/github.com_matan-b_ceph_dbebf1a3db6db7b5579464ee715edcf1b3e9cce2/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/matan-b/ceph.git', 'suite_sha1': 'dbebf1a3db6db7b5579464ee715edcf1b3e9cce2', 'targets': {'smithi188.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIpeqYZWsRy97+M4/mZVh8lxMGz6ycVQE3TYoZJrvY+0qZgaOiu4wdStLjwzIZzzioaLyYYgq8pKomtSa9o79FY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ["rbd/test_librbd_python.sh -m 'not skip_if_crimson'"]}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-11_13:56:22', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-11T14:35:52.896 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-11T14:35:52.944 DEBUG:teuthology.orchestra.run.smithi188:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T14:35:53.009 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-11T14:35:53.009 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T14:35:53.065 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T14:35:53.066 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_1 2024-09-11T14:35:53.120 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T14:35:53.121 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T14:35:53.121 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-11T14:35:53.121 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T14:35:53.121 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T14:35:53.121 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-09-11 14:35:35.554809921 +0000 2024-09-11T14:35:53.121 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-09-11 14:33:30.256596335 +0000 2024-09-11T14:35:53.121 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-09-11 14:33:30.256596335 +0000 2024-09-11T14:35:53.121 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-09-11 14:33:30.256596335 +0000 2024-09-11T14:35:53.121 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T14:35:53.184 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-09-11T14:35:53.184 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-09-11T14:35:53.184 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000269392 s, 1.9 MB/s 2024-09-11T14:35:53.186 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T14:35:53.242 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_2 2024-09-11T14:35:53.298 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T14:35:53.298 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T14:35:53.298 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-11T14:35:53.298 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T14:35:53.299 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T14:35:53.299 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-09-11 14:35:35.555809902 +0000 2024-09-11T14:35:53.299 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-09-11 14:33:30.595589952 +0000 2024-09-11T14:35:53.299 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-09-11 14:33:30.595589952 +0000 2024-09-11T14:35:53.299 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-09-11 14:33:30.595589952 +0000 2024-09-11T14:35:53.299 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T14:35:53.361 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-09-11T14:35:53.361 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-09-11T14:35:53.361 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000180215 s, 2.8 MB/s 2024-09-11T14:35:53.362 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T14:35:53.418 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_3 2024-09-11T14:35:53.474 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T14:35:53.474 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T14:35:53.474 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-11T14:35:53.474 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T14:35:53.474 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T14:35:53.474 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-09-11 14:35:35.555809902 +0000 2024-09-11T14:35:53.474 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-09-11 14:33:30.931583625 +0000 2024-09-11T14:35:53.474 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-09-11 14:33:30.931583625 +0000 2024-09-11T14:35:53.475 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-09-11 14:33:30.931583625 +0000 2024-09-11T14:35:53.475 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T14:35:53.537 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-09-11T14:35:53.537 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-09-11T14:35:53.537 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000163108 s, 3.1 MB/s 2024-09-11T14:35:53.539 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T14:35:53.595 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_4 2024-09-11T14:35:53.650 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T14:35:53.650 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T14:35:53.650 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-11T14:35:53.651 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T14:35:53.651 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T14:35:53.651 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-09-11 14:35:35.556809883 +0000 2024-09-11T14:35:53.651 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-09-11 14:33:31.245577713 +0000 2024-09-11T14:35:53.651 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-09-11 14:33:31.245577713 +0000 2024-09-11T14:35:53.651 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-09-11 14:33:31.245577713 +0000 2024-09-11T14:35:53.651 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T14:35:53.713 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-09-11T14:35:53.713 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-09-11T14:35:53.713 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000214632 s, 2.4 MB/s 2024-09-11T14:35:53.714 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T14:35:53.770 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T14:35:53.770 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi188.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-11T14:35:53.770 INFO:tasks.ceph:Generating config... 2024-09-11T14:35:53.771 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-09-11T14:35:53.772 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-11T14:35:53.772 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-11T14:35:53.772 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-11T14:35:53.772 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-09-11T14:35:53.772 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-11T14:35:53.772 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-11T14:35:53.772 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-11T14:35:53.772 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-09-11T14:35:53.772 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-11T14:35:53.772 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-09-11T14:35:53.772 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-09-11T14:35:53.772 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-11T14:35:53.773 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-11T14:35:53.773 INFO:tasks.ceph:[osd] debug seastore = 20 2024-09-11T14:35:53.773 INFO:tasks.ceph:[osd] debug seastore backref = 20 2024-09-11T14:35:53.773 INFO:tasks.ceph:[osd] debug seastore cache = 20 2024-09-11T14:35:53.773 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2024-09-11T14:35:53.773 INFO:tasks.ceph:[osd] debug seastore device = 20 2024-09-11T14:35:53.773 INFO:tasks.ceph:[osd] debug seastore epm = 20 2024-09-11T14:35:53.773 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2024-09-11T14:35:53.773 INFO:tasks.ceph:[osd] debug seastore journal = 20 2024-09-11T14:35:53.773 INFO:tasks.ceph:[osd] debug seastore lba = 20 2024-09-11T14:35:53.773 INFO:tasks.ceph:[osd] debug seastore odata = 20 2024-09-11T14:35:53.773 INFO:tasks.ceph:[osd] debug seastore ompap = 20 2024-09-11T14:35:53.773 INFO:tasks.ceph:[osd] debug seastore onode = 20 2024-09-11T14:35:53.773 INFO:tasks.ceph:[osd] debug seastore t = 20 2024-09-11T14:35:53.774 INFO:tasks.ceph:[osd] debug seastore tm = 20 2024-09-11T14:35:53.774 INFO:tasks.ceph:[osd] osd objectstore = seastore 2024-09-11T14:35:53.774 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-11T14:35:53.774 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-09-11T14:35:53.774 INFO:tasks.ceph:[global] ms client mode = crc 2024-09-11T14:35:53.774 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-09-11T14:35:53.774 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-09-11T14:35:53.774 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-09-11T14:35:53.774 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-09-11T14:35:53.774 INFO:tasks.ceph:[global] ms service mode = crc 2024-09-11T14:35:53.774 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-11T14:35:53.774 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-11T14:35:53.774 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-1 2024-09-11T14:35:53.774 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 2-3 2024-09-11T14:35:53.775 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 4-5 2024-09-11T14:35:53.775 INFO:tasks.ceph:Setting up mon.a... 2024-09-11T14:35:53.775 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-11T14:35:53.868 INFO:teuthology.orchestra.run.smithi188.stdout:creating /etc/ceph/ceph.keyring 2024-09-11T14:35:53.871 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-11T14:35:53.922 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T14:35:53.947 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.188')] 2024-09-11T14:35:53.947 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.188', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}} 2024-09-11T14:35:53.948 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-11T14:35:53.948 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T14:35:54.003 DEBUG:teuthology.orchestra.run.smithi188:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.188 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:35:54.094 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:35:54.092+0000 7f5719c90f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:54.094 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:35:54.092+0000 7f5719c90f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:54.094 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:35:54.093+0000 7f5719c90f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:54.095 INFO:teuthology.orchestra.run.smithi188.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:35:54.095 INFO:teuthology.orchestra.run.smithi188.stdout:monmaptool: generated fsid 1c448697-f86d-4ae7-8a96-5392aad74a40 2024-09-11T14:35:54.095 INFO:teuthology.orchestra.run.smithi188.stdout:setting min_mon_release = quincy 2024-09-11T14:35:54.095 INFO:teuthology.orchestra.run.smithi188.stdout:epoch 0 2024-09-11T14:35:54.095 INFO:teuthology.orchestra.run.smithi188.stdout:fsid 1c448697-f86d-4ae7-8a96-5392aad74a40 2024-09-11T14:35:54.095 INFO:teuthology.orchestra.run.smithi188.stdout:last_changed 2024-09-11T14:35:54.094116+0000 2024-09-11T14:35:54.095 INFO:teuthology.orchestra.run.smithi188.stdout:created 2024-09-11T14:35:54.094116+0000 2024-09-11T14:35:54.095 INFO:teuthology.orchestra.run.smithi188.stdout:min_mon_release 17 (quincy) 2024-09-11T14:35:54.095 INFO:teuthology.orchestra.run.smithi188.stdout:election_strategy: 1 2024-09-11T14:35:54.095 INFO:teuthology.orchestra.run.smithi188.stdout:0: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.a 2024-09-11T14:35:54.095 INFO:teuthology.orchestra.run.smithi188.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-09-11T14:35:54.096 DEBUG:teuthology.orchestra.run.smithi188:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T14:35:54.111 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 1c448697-f86d-4ae7-8a96-5392aad74a40... 2024-09-11T14:35:54.112 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-11T14:35:54.193 INFO:teuthology.orchestra.run.smithi188.stdout:[global] 2024-09-11T14:35:54.194 INFO:teuthology.orchestra.run.smithi188.stdout: chdir = "" 2024-09-11T14:35:54.194 INFO:teuthology.orchestra.run.smithi188.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T14:35:54.194 INFO:teuthology.orchestra.run.smithi188.stdout: auth supported = cephx 2024-09-11T14:35:54.194 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:54.194 INFO:teuthology.orchestra.run.smithi188.stdout: filestore xattr use omap = true 2024-09-11T14:35:54.194 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:54.194 INFO:teuthology.orchestra.run.smithi188.stdout: mon clock drift allowed = 1.000 2024-09-11T14:35:54.194 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:54.194 INFO:teuthology.orchestra.run.smithi188.stdout: osd crush chooseleaf type = 0 2024-09-11T14:35:54.194 INFO:teuthology.orchestra.run.smithi188.stdout: auth debug = true 2024-09-11T14:35:54.194 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:54.194 INFO:teuthology.orchestra.run.smithi188.stdout: ms die on old message = true 2024-09-11T14:35:54.194 INFO:teuthology.orchestra.run.smithi188.stdout: ms die on bug = true 2024-09-11T14:35:54.194 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:54.195 INFO:teuthology.orchestra.run.smithi188.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T14:35:54.195 INFO:teuthology.orchestra.run.smithi188.stdout: mon pg warn max object skew = 0 2024-09-11T14:35:54.195 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:54.195 INFO:teuthology.orchestra.run.smithi188.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T14:35:54.195 INFO:teuthology.orchestra.run.smithi188.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T14:35:54.195 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:54.195 INFO:teuthology.orchestra.run.smithi188.stdout: osd pool default size = 2 2024-09-11T14:35:54.195 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:54.195 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd allow primary affinity = true 2024-09-11T14:35:54.195 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd allow pg remap = true 2024-09-11T14:35:54.195 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on legacy crush tunables = false 2024-09-11T14:35:54.195 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on crush straw calc version zero = false 2024-09-11T14:35:54.195 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on no sortbitwise = false 2024-09-11T14:35:54.196 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on osd down out interval zero = false 2024-09-11T14:35:54.196 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on too few osds = false 2024-09-11T14:35:54.196 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T14:35:54.196 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T14:35:54.196 INFO:teuthology.orchestra.run.smithi188.stdout: mon_allow_pool_size_one = true 2024-09-11T14:35:54.196 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:54.196 INFO:teuthology.orchestra.run.smithi188.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T14:35:54.196 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:54.196 INFO:teuthology.orchestra.run.smithi188.stdout: osd default data pool replay window = 5 2024-09-11T14:35:54.196 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:54.196 INFO:teuthology.orchestra.run.smithi188.stdout: mon allow pool delete = true 2024-09-11T14:35:54.196 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:54.196 INFO:teuthology.orchestra.run.smithi188.stdout: mon cluster log file level = debug 2024-09-11T14:35:54.196 INFO:teuthology.orchestra.run.smithi188.stdout: debug asserts on shutdown = true 2024-09-11T14:35:54.197 INFO:teuthology.orchestra.run.smithi188.stdout: mon health detail to clog = false 2024-09-11T14:35:54.197 INFO:teuthology.orchestra.run.smithi188.stdout: mon host = 172.21.15.188 2024-09-11T14:35:54.197 INFO:teuthology.orchestra.run.smithi188.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-09-11T14:35:54.197 INFO:teuthology.orchestra.run.smithi188.stdout: ms client mode = crc 2024-09-11T14:35:54.197 INFO:teuthology.orchestra.run.smithi188.stdout: ms cluster mode = crc 2024-09-11T14:35:54.197 INFO:teuthology.orchestra.run.smithi188.stdout: ms mon client mode = crc 2024-09-11T14:35:54.197 INFO:teuthology.orchestra.run.smithi188.stdout: ms mon cluster mode = crc 2024-09-11T14:35:54.197 INFO:teuthology.orchestra.run.smithi188.stdout: ms mon service mode = crc 2024-09-11T14:35:54.197 INFO:teuthology.orchestra.run.smithi188.stdout: ms service mode = crc 2024-09-11T14:35:54.197 INFO:teuthology.orchestra.run.smithi188.stdout: fsid = 1c448697-f86d-4ae7-8a96-5392aad74a40 2024-09-11T14:35:54.197 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:54.197 INFO:teuthology.orchestra.run.smithi188.stdout:[osd] 2024-09-11T14:35:54.197 INFO:teuthology.orchestra.run.smithi188.stdout: osd journal size = 100 2024-09-11T14:35:54.197 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:54.198 INFO:teuthology.orchestra.run.smithi188.stdout: osd scrub load threshold = 5.0 2024-09-11T14:35:54.198 INFO:teuthology.orchestra.run.smithi188.stdout: osd scrub max interval = 600 2024-09-11T14:35:54.198 INFO:teuthology.orchestra.run.smithi188.stdout: osd mclock profile = high_recovery_ops 2024-09-11T14:35:54.198 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:54.198 INFO:teuthology.orchestra.run.smithi188.stdout: osd recover clone overlap = true 2024-09-11T14:35:54.198 INFO:teuthology.orchestra.run.smithi188.stdout: osd recovery max chunk = 1048576 2024-09-11T14:35:54.198 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:54.198 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug shutdown = true 2024-09-11T14:35:54.198 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug op order = true 2024-09-11T14:35:54.198 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug verify stray on activate = true 2024-09-11T14:35:54.198 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug trim objects = true 2024-09-11T14:35:54.198 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:54.198 INFO:teuthology.orchestra.run.smithi188.stdout: osd open classes on start = true 2024-09-11T14:35:54.198 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug pg log writeout = true 2024-09-11T14:35:54.198 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:54.198 INFO:teuthology.orchestra.run.smithi188.stdout: osd deep scrub update digest min age = 30 2024-09-11T14:35:54.199 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:54.199 INFO:teuthology.orchestra.run.smithi188.stdout: osd map max advance = 10 2024-09-11T14:35:54.199 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:54.199 INFO:teuthology.orchestra.run.smithi188.stdout: journal zero on create = true 2024-09-11T14:35:54.199 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:54.199 INFO:teuthology.orchestra.run.smithi188.stdout: filestore ondisk finisher threads = 3 2024-09-11T14:35:54.199 INFO:teuthology.orchestra.run.smithi188.stdout: filestore apply finisher threads = 3 2024-09-11T14:35:54.199 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:54.199 INFO:teuthology.orchestra.run.smithi188.stdout: bdev debug aio = true 2024-09-11T14:35:54.199 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug misdirected ops = true 2024-09-11T14:35:54.199 INFO:teuthology.orchestra.run.smithi188.stdout: debug monc = 20 2024-09-11T14:35:54.199 INFO:teuthology.orchestra.run.smithi188.stdout: crimson alien thread cpu cores = 6-7 2024-09-11T14:35:54.199 INFO:teuthology.orchestra.run.smithi188.stdout: crimson osd obc lru size = 10 2024-09-11T14:35:54.199 INFO:teuthology.orchestra.run.smithi188.stdout: debug ms = 1 2024-09-11T14:35:54.199 INFO:teuthology.orchestra.run.smithi188.stdout: debug osd = 20 2024-09-11T14:35:54.200 INFO:teuthology.orchestra.run.smithi188.stdout: debug seastore = 20 2024-09-11T14:35:54.200 INFO:teuthology.orchestra.run.smithi188.stdout: debug seastore backref = 20 2024-09-11T14:35:54.200 INFO:teuthology.orchestra.run.smithi188.stdout: debug seastore cache = 20 2024-09-11T14:35:54.200 INFO:teuthology.orchestra.run.smithi188.stdout: debug seastore cleaner = 20 2024-09-11T14:35:54.200 INFO:teuthology.orchestra.run.smithi188.stdout: debug seastore device = 20 2024-09-11T14:35:54.200 INFO:teuthology.orchestra.run.smithi188.stdout: debug seastore epm = 20 2024-09-11T14:35:54.200 INFO:teuthology.orchestra.run.smithi188.stdout: debug seastore fixedkv tree = 20 2024-09-11T14:35:54.200 INFO:teuthology.orchestra.run.smithi188.stdout: debug seastore journal = 20 2024-09-11T14:35:54.200 INFO:teuthology.orchestra.run.smithi188.stdout: debug seastore lba = 20 2024-09-11T14:35:54.200 INFO:teuthology.orchestra.run.smithi188.stdout: debug seastore odata = 20 2024-09-11T14:35:54.200 INFO:teuthology.orchestra.run.smithi188.stdout: debug seastore ompap = 20 2024-09-11T14:35:54.200 INFO:teuthology.orchestra.run.smithi188.stdout: debug seastore onode = 20 2024-09-11T14:35:54.200 INFO:teuthology.orchestra.run.smithi188.stdout: debug seastore t = 20 2024-09-11T14:35:54.200 INFO:teuthology.orchestra.run.smithi188.stdout: debug seastore tm = 20 2024-09-11T14:35:54.200 INFO:teuthology.orchestra.run.smithi188.stdout: osd objectstore = seastore 2024-09-11T14:35:54.201 INFO:teuthology.orchestra.run.smithi188.stdout: osd shutdown pgref assert = True 2024-09-11T14:35:54.201 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:54.201 INFO:teuthology.orchestra.run.smithi188.stdout:[mgr] 2024-09-11T14:35:54.201 INFO:teuthology.orchestra.run.smithi188.stdout: debug ms = 1 2024-09-11T14:35:54.201 INFO:teuthology.orchestra.run.smithi188.stdout: debug mgr = 20 2024-09-11T14:35:54.201 INFO:teuthology.orchestra.run.smithi188.stdout: debug mon = 20 2024-09-11T14:35:54.201 INFO:teuthology.orchestra.run.smithi188.stdout: debug auth = 20 2024-09-11T14:35:54.201 INFO:teuthology.orchestra.run.smithi188.stdout: mon reweight min pgs per osd = 4 2024-09-11T14:35:54.201 INFO:teuthology.orchestra.run.smithi188.stdout: mon reweight min bytes per osd = 10 2024-09-11T14:35:54.201 INFO:teuthology.orchestra.run.smithi188.stdout: mgr/telemetry/nag = false 2024-09-11T14:35:54.201 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:54.201 INFO:teuthology.orchestra.run.smithi188.stdout:[mon] 2024-09-11T14:35:54.201 INFO:teuthology.orchestra.run.smithi188.stdout: debug ms = 1 2024-09-11T14:35:54.201 INFO:teuthology.orchestra.run.smithi188.stdout: debug mon = 20 2024-09-11T14:35:54.201 INFO:teuthology.orchestra.run.smithi188.stdout: debug paxos = 20 2024-09-11T14:35:54.202 INFO:teuthology.orchestra.run.smithi188.stdout: debug auth = 20 2024-09-11T14:35:54.202 INFO:teuthology.orchestra.run.smithi188.stdout: mon data avail warn = 5 2024-09-11T14:35:54.202 INFO:teuthology.orchestra.run.smithi188.stdout: mon mgr mkfs grace = 240 2024-09-11T14:35:54.202 INFO:teuthology.orchestra.run.smithi188.stdout: mon reweight min pgs per osd = 4 2024-09-11T14:35:54.202 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd reporter subtree level = osd 2024-09-11T14:35:54.202 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd prime pg temp = true 2024-09-11T14:35:54.202 INFO:teuthology.orchestra.run.smithi188.stdout: mon reweight min bytes per osd = 10 2024-09-11T14:35:54.202 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:54.202 INFO:teuthology.orchestra.run.smithi188.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T14:35:54.202 INFO:teuthology.orchestra.run.smithi188.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T14:35:54.202 INFO:teuthology.orchestra.run.smithi188.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T14:35:54.202 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:54.202 INFO:teuthology.orchestra.run.smithi188.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T14:35:54.202 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T14:35:54.202 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T14:35:54.203 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:54.203 INFO:teuthology.orchestra.run.smithi188.stdout: # 1m isn't quite enough 2024-09-11T14:35:54.203 INFO:teuthology.orchestra.run.smithi188.stdout: mon_down_mkfs_grace = 2m 2024-09-11T14:35:54.203 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:54.203 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_filestore_osds = false 2024-09-11T14:35:54.203 INFO:teuthology.orchestra.run.smithi188.stdout: mon min osdmap epochs = 50 2024-09-11T14:35:54.203 INFO:teuthology.orchestra.run.smithi188.stdout: mon osdmap full prune interval = 2 2024-09-11T14:35:54.203 INFO:teuthology.orchestra.run.smithi188.stdout: mon osdmap full prune min = 15 2024-09-11T14:35:54.203 INFO:teuthology.orchestra.run.smithi188.stdout: mon osdmap full prune txsize = 2 2024-09-11T14:35:54.203 INFO:teuthology.orchestra.run.smithi188.stdout: paxos service trim min = 10 2024-09-11T14:35:54.203 INFO:teuthology.orchestra.run.smithi188.stdout: osd pool default crimson = True 2024-09-11T14:35:54.203 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:35:54.203 INFO:teuthology.orchestra.run.smithi188.stdout:[client] 2024-09-11T14:35:54.203 INFO:teuthology.orchestra.run.smithi188.stdout: rgw cache enabled = true 2024-09-11T14:35:54.203 INFO:teuthology.orchestra.run.smithi188.stdout: rgw enable ops log = true 2024-09-11T14:35:54.204 INFO:teuthology.orchestra.run.smithi188.stdout: rgw enable usage log = true 2024-09-11T14:35:54.204 INFO:teuthology.orchestra.run.smithi188.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T14:35:54.204 INFO:teuthology.orchestra.run.smithi188.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T14:35:54.204 INFO:teuthology.orchestra.run.smithi188.stdout:[mon.a] 2024-09-11T14:35:54.204 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.0] 2024-09-11T14:35:54.204 INFO:teuthology.orchestra.run.smithi188.stdout: crimson seastar cpu cores = 0-1 2024-09-11T14:35:54.204 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.1] 2024-09-11T14:35:54.204 INFO:teuthology.orchestra.run.smithi188.stdout: crimson seastar cpu cores = 2-3 2024-09-11T14:35:54.204 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.2] 2024-09-11T14:35:54.204 INFO:teuthology.orchestra.run.smithi188.stdout: crimson seastar cpu cores = 4-5 2024-09-11T14:35:54.205 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-11T14:35:54.205 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-11T14:35:54.311 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-11T14:35:54.312 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-11T14:35:54.312 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-11T14:35:54.327 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-11T14:35:54.327 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-11T14:35:54.382 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi188.front.sepia.ceph.com 2024-09-11T14:35:54.383 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-11T14:35:54.383 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-11T14:35:54.383 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T14:35:54.458 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-11T14:35:54.458 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:35:54.513 INFO:tasks.ceph:Setting up mon nodes... 2024-09-11T14:35:54.513 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-11T14:35:54.513 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-11T14:35:54.612 INFO:teuthology.orchestra.run.smithi188.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-11T14:35:54.615 INFO:tasks.ceph:Setting up mds nodes... 2024-09-11T14:35:54.615 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-11T14:35:54.615 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-11T14:35:54.665 INFO:teuthology.orchestra.run.smithi188.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-11T14:35:54.678 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-11T14:35:54.678 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi188.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-11T14:35:54.678 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-11T14:35:54.703 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T14:35:54.703 INFO:tasks.ceph:role: osd.0 2024-09-11T14:35:54.703 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi188.front.sepia.ceph.com 2024-09-11T14:35:54.703 DEBUG:teuthology.orchestra.run.smithi188:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-11T14:35:54.775 INFO:teuthology.orchestra.run.smithi188.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T14:35:54.775 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T14:35:54.775 INFO:teuthology.orchestra.run.smithi188.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T14:35:54.775 INFO:teuthology.orchestra.run.smithi188.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T14:35:54.775 INFO:teuthology.orchestra.run.smithi188.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T14:35:54.775 INFO:teuthology.orchestra.run.smithi188.stdout: = sunit=0 swidth=0 blks 2024-09-11T14:35:54.775 INFO:teuthology.orchestra.run.smithi188.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T14:35:54.775 INFO:teuthology.orchestra.run.smithi188.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T14:35:54.775 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T14:35:54.775 INFO:teuthology.orchestra.run.smithi188.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T14:35:55.291 INFO:teuthology.orchestra.run.smithi188.stdout:Discarding blocks...Done. 2024-09-11T14:35:55.293 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi188.front.sepia.ceph.com -o noatime 2024-09-11T14:35:55.293 DEBUG:teuthology.orchestra.run.smithi188:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-11T14:35:55.398 DEBUG:teuthology.orchestra.run.smithi188:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-11T14:35:55.432 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-11T14:35:55.497 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T14:35:55.498 INFO:tasks.ceph:role: osd.1 2024-09-11T14:35:55.498 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi188.front.sepia.ceph.com 2024-09-11T14:35:55.498 DEBUG:teuthology.orchestra.run.smithi188:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-11T14:35:55.570 INFO:teuthology.orchestra.run.smithi188.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T14:35:55.570 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T14:35:55.570 INFO:teuthology.orchestra.run.smithi188.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T14:35:55.570 INFO:teuthology.orchestra.run.smithi188.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T14:35:55.570 INFO:teuthology.orchestra.run.smithi188.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T14:35:55.570 INFO:teuthology.orchestra.run.smithi188.stdout: = sunit=0 swidth=0 blks 2024-09-11T14:35:55.571 INFO:teuthology.orchestra.run.smithi188.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T14:35:55.571 INFO:teuthology.orchestra.run.smithi188.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T14:35:55.571 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T14:35:55.571 INFO:teuthology.orchestra.run.smithi188.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T14:35:56.021 INFO:teuthology.orchestra.run.smithi188.stdout:Discarding blocks...Done. 2024-09-11T14:35:56.023 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi188.front.sepia.ceph.com -o noatime 2024-09-11T14:35:56.023 DEBUG:teuthology.orchestra.run.smithi188:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-11T14:35:56.113 DEBUG:teuthology.orchestra.run.smithi188:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-11T14:35:56.143 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-11T14:35:56.208 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T14:35:56.208 INFO:tasks.ceph:role: osd.2 2024-09-11T14:35:56.208 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi188.front.sepia.ceph.com 2024-09-11T14:35:56.208 DEBUG:teuthology.orchestra.run.smithi188:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-11T14:35:56.281 INFO:teuthology.orchestra.run.smithi188.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T14:35:56.281 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T14:35:56.281 INFO:teuthology.orchestra.run.smithi188.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T14:35:56.281 INFO:teuthology.orchestra.run.smithi188.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T14:35:56.281 INFO:teuthology.orchestra.run.smithi188.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T14:35:56.281 INFO:teuthology.orchestra.run.smithi188.stdout: = sunit=0 swidth=0 blks 2024-09-11T14:35:56.281 INFO:teuthology.orchestra.run.smithi188.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T14:35:56.281 INFO:teuthology.orchestra.run.smithi188.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T14:35:56.281 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T14:35:56.281 INFO:teuthology.orchestra.run.smithi188.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T14:35:56.765 INFO:teuthology.orchestra.run.smithi188.stdout:Discarding blocks...Done. 2024-09-11T14:35:56.767 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi188.front.sepia.ceph.com -o noatime 2024-09-11T14:35:56.767 DEBUG:teuthology.orchestra.run.smithi188:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-11T14:35:56.909 DEBUG:teuthology.orchestra.run.smithi188:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-11T14:35:56.937 DEBUG:teuthology.orchestra.run.smithi188:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:35:57.234 INFO:teuthology.orchestra.run.smithi188.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:35:57.235 INFO:teuthology.orchestra.run.smithi188.stderr:INFO 2024-09-11 14:35:57,235 seastar - Reactor backend: linux-aio 2024-09-11T14:35:57.246 INFO:teuthology.orchestra.run.smithi188.stderr:==25508==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:35:57.364 INFO:teuthology.orchestra.run.smithi188.stderr:INFO 2024-09-11 14:35:57,363 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-11T14:35:57.364 INFO:teuthology.orchestra.run.smithi188.stderr:INFO 2024-09-11 14:35:57,363 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 0-1 2024-09-11T14:35:57.364 INFO:teuthology.orchestra.run.smithi188.stderr:DEBUG 2024-09-11 14:35:57,364 [shard 0:main] seastar - reactor::drain 2024-09-11T14:35:57.888 INFO:teuthology.orchestra.run.smithi188.stderr:----------------------------------------------------- 2024-09-11T14:35:57.888 INFO:teuthology.orchestra.run.smithi188.stderr:Suppressions used: 2024-09-11T14:35:57.888 INFO:teuthology.orchestra.run.smithi188.stderr: count bytes template 2024-09-11T14:35:57.888 INFO:teuthology.orchestra.run.smithi188.stderr: 1 8 MallocExtension::Register 2024-09-11T14:35:57.888 INFO:teuthology.orchestra.run.smithi188.stderr:----------------------------------------------------- 2024-09-11T14:35:57.888 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-09-11T14:35:57.945 INFO:teuthology.orchestra.run.smithi188.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:35:57.949 INFO:teuthology.orchestra.run.smithi188.stderr:INFO 2024-09-11 14:35:57,948 seastar - Reactor backend: linux-aio 2024-09-11T14:35:57.957 INFO:teuthology.orchestra.run.smithi188.stderr:==25506==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:35:58.240 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x119673 0x11d39b 0x41608 0x41bbe 0x41c92 0x41f68 0x4214e 0x3ef7b 0xfd531 0xba2d45f 0xb9d2631 0x9b01071 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:35:58.240 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb0f9bd7e 0xffffffffb0f9c617 0xffffffffb0fa050b 0xffffffffb0fa0b9d 0xffffffffb0f6cd4d 0xffffffffb18cd45f 0xffffffffb1a00130 2024-09-11T14:35:58.537 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x5549f82 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:35:58.538 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 2024-09-11T14:35:58.595 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x133d18 0xdd332 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:35:58.595 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 2024-09-11T14:35:58.623 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x4abef 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:35:58.624 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 2024-09-11T14:35:58.681 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x133d0c 0xdd332 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:35:58.681 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 2024-09-11T14:35:58.702 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x554928f 0x9e3b019 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:35:58.702 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 2024-09-11T14:35:58.789 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 230 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x10d76c 0xdd31a 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:35:58.789 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 2024-09-11T14:35:59.070 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x10d75f 0xdd31a 0x4ab1c 0xb85ba65 0xb85bceb 0xba2da49 0xba312a0 0xb9e403b 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93195d6 0x931a07a 0x931a0fd 0x931a182 0x931a284 0x931a37c 0x931a402 0x931a4bb 0x931a671 0x931a7a9 0x931a824 0x931a8a9 0x931aa04 0x931ab1e 0x931aba3 0x931ac81 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:35:59.070 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 2024-09-11T14:35:59.229 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x9e3b026 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:35:59.230 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 2024-09-11T14:35:59.287 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x53b10 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:35:59.287 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 2024-09-11T14:36:00.234 INFO:teuthology.orchestra.run.smithi188.stderr:----------------------------------------------------- 2024-09-11T14:36:00.235 INFO:teuthology.orchestra.run.smithi188.stderr:Suppressions used: 2024-09-11T14:36:00.235 INFO:teuthology.orchestra.run.smithi188.stderr: count bytes template 2024-09-11T14:36:00.235 INFO:teuthology.orchestra.run.smithi188.stderr: 1 8 MallocExtension::Register 2024-09-11T14:36:00.235 INFO:teuthology.orchestra.run.smithi188.stderr:----------------------------------------------------- 2024-09-11T14:36:00.235 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-09-11T14:36:00.261 DEBUG:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-11T14:36:00.286 DEBUG:teuthology.orchestra.run.smithi188:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:36:00.567 INFO:teuthology.orchestra.run.smithi188.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:36:00.568 INFO:teuthology.orchestra.run.smithi188.stderr:INFO 2024-09-11 14:36:00,568 seastar - Reactor backend: linux-aio 2024-09-11T14:36:00.573 INFO:teuthology.orchestra.run.smithi188.stderr:==25555==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:36:00.692 INFO:teuthology.orchestra.run.smithi188.stderr:INFO 2024-09-11 14:36:00,692 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-11T14:36:00.693 INFO:teuthology.orchestra.run.smithi188.stderr:INFO 2024-09-11 14:36:00,692 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 2-3 2024-09-11T14:36:00.693 INFO:teuthology.orchestra.run.smithi188.stderr:DEBUG 2024-09-11 14:36:00,693 [shard 0:main] seastar - reactor::drain 2024-09-11T14:36:01.192 INFO:teuthology.orchestra.run.smithi188.stderr:----------------------------------------------------- 2024-09-11T14:36:01.193 INFO:teuthology.orchestra.run.smithi188.stderr:Suppressions used: 2024-09-11T14:36:01.193 INFO:teuthology.orchestra.run.smithi188.stderr: count bytes template 2024-09-11T14:36:01.193 INFO:teuthology.orchestra.run.smithi188.stderr: 1 8 MallocExtension::Register 2024-09-11T14:36:01.193 INFO:teuthology.orchestra.run.smithi188.stderr:----------------------------------------------------- 2024-09-11T14:36:01.193 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-09-11T14:36:01.255 INFO:teuthology.orchestra.run.smithi188.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:36:01.258 INFO:teuthology.orchestra.run.smithi188.stderr:INFO 2024-09-11 14:36:01,258 seastar - Reactor backend: linux-aio 2024-09-11T14:36:01.273 INFO:teuthology.orchestra.run.smithi188.stderr:==25553==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:36:01.558 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x161c3f 0xc928c 0xc9e39 0x360e4e8 0x3b92070 0x3b92268 0xb9f768b 0xb9fb0f9 0xba3eb1c 0xb9dfda2 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:36:01.558 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 2024-09-11T14:36:01.732 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0xb9f0c44 0xba2f04a 0xb9d3249 0xba2fa98 0xba2fcdf 0xba2fd8f 0xb9df75d 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b024e4 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:36:01.732 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 2024-09-11T14:36:01.853 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x554936f 0x9e3b019 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:36:01.853 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 2024-09-11T14:36:01.911 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x9e3aff4 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:36:01.911 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 2024-09-11T14:36:01.928 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x10d778 0xdd31a 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:36:01.928 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 2024-09-11T14:36:01.985 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0xdd339 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:36:01.986 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 2024-09-11T14:36:02.018 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x5549680 0x9e3b02e 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:36:02.019 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 2024-09-11T14:36:02.093 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 230 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x4abc8 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:36:02.093 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 2024-09-11T14:36:02.539 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x5549aed 0x9e3b03b 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:36:02.540 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 2024-09-11T14:36:02.597 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x10d778 0xdd31a 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:36:02.597 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 2024-09-11T14:36:03.545 INFO:teuthology.orchestra.run.smithi188.stderr:----------------------------------------------------- 2024-09-11T14:36:03.545 INFO:teuthology.orchestra.run.smithi188.stderr:Suppressions used: 2024-09-11T14:36:03.545 INFO:teuthology.orchestra.run.smithi188.stderr: count bytes template 2024-09-11T14:36:03.545 INFO:teuthology.orchestra.run.smithi188.stderr: 1 8 MallocExtension::Register 2024-09-11T14:36:03.545 INFO:teuthology.orchestra.run.smithi188.stderr:----------------------------------------------------- 2024-09-11T14:36:03.545 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-09-11T14:36:03.571 DEBUG:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-11T14:36:03.596 DEBUG:teuthology.orchestra.run.smithi188:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:36:03.878 INFO:teuthology.orchestra.run.smithi188.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:36:03.879 INFO:teuthology.orchestra.run.smithi188.stderr:INFO 2024-09-11 14:36:03,879 seastar - Reactor backend: linux-aio 2024-09-11T14:36:03.884 INFO:teuthology.orchestra.run.smithi188.stderr:==25600==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:36:04.005 INFO:teuthology.orchestra.run.smithi188.stderr:INFO 2024-09-11 14:36:04,004 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-11T14:36:04.005 INFO:teuthology.orchestra.run.smithi188.stderr:INFO 2024-09-11 14:36:04,004 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 4-5 2024-09-11T14:36:04.005 INFO:teuthology.orchestra.run.smithi188.stderr:DEBUG 2024-09-11 14:36:04,005 [shard 0:main] seastar - reactor::drain 2024-09-11T14:36:04.514 INFO:teuthology.orchestra.run.smithi188.stderr:----------------------------------------------------- 2024-09-11T14:36:04.514 INFO:teuthology.orchestra.run.smithi188.stderr:Suppressions used: 2024-09-11T14:36:04.514 INFO:teuthology.orchestra.run.smithi188.stderr: count bytes template 2024-09-11T14:36:04.515 INFO:teuthology.orchestra.run.smithi188.stderr: 1 8 MallocExtension::Register 2024-09-11T14:36:04.515 INFO:teuthology.orchestra.run.smithi188.stderr:----------------------------------------------------- 2024-09-11T14:36:04.515 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-09-11T14:36:04.571 INFO:teuthology.orchestra.run.smithi188.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:36:04.574 INFO:teuthology.orchestra.run.smithi188.stderr:INFO 2024-09-11 14:36:04,574 seastar - Reactor backend: linux-aio 2024-09-11T14:36:04.591 INFO:teuthology.orchestra.run.smithi188.stderr:==25598==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:36:04.871 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x360e4c5 0x3b92070 0x3b92268 0xb9f075a 0xb9f1bfa 0xb9f1e74 0xb9f201c 0xba0999f 0xba38a91 0xba390d8 0xb9e038f 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:36:04.871 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 2024-09-11T14:36:04.954 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x162c49 0x4162a 0x41bbe 0x41c92 0x41f68 0x4214e 0x3ef7b 0xfd531 0xb85ab4d 0xb85ac00 0xb85ad16 0xb85adf6 0xb85b076 0xb85b621 0xb85ba2c 0xb85bceb 0xb85c7a8 0xba2d476 0xb9d2631 0x9b01657 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:36:04.954 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb12a5d17 0xffffffffb0fba036 0xffffffffb0fbdce1 0xffffffffb0fbee42 0xffffffffb0fbf6b7 0xffffffffb0fe4853 0xffffffffb0f8ebde 0xffffffffb0f96a3b 0xffffffffb0f96ebc 0xffffffffb0c8c884 0xffffffffb18d1d62 0xffffffffb1a00bb2 2024-09-11T14:36:05.042 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x124f33 0x421f3 0x3ecea 0xfc9b1 0x38833ab 0x3883747 0x9b00f65 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:36:05.042 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 2024-09-11T14:36:05.147 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x4ab7c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:36:05.147 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 2024-09-11T14:36:05.205 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x554a0cc 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:36:05.205 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 2024-09-11T14:36:05.209 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x554a153 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:36:05.210 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 2024-09-11T14:36:05.267 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x5549aed 0x9e3b03b 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:36:05.267 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 2024-09-11T14:36:05.312 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x4ab09 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:36:05.313 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 2024-09-11T14:36:05.375 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 230 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x5549b26 0x9e3b03b 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:36:05.375 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 2024-09-11T14:36:05.864 INFO:teuthology.orchestra.run.smithi188.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x10d75f 0xdd31a 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:36:05.864 INFO:teuthology.orchestra.run.smithi188.stderr:kernel callstack: 2024-09-11T14:36:06.848 INFO:teuthology.orchestra.run.smithi188.stderr:----------------------------------------------------- 2024-09-11T14:36:06.848 INFO:teuthology.orchestra.run.smithi188.stderr:Suppressions used: 2024-09-11T14:36:06.848 INFO:teuthology.orchestra.run.smithi188.stderr: count bytes template 2024-09-11T14:36:06.849 INFO:teuthology.orchestra.run.smithi188.stderr: 1 8 MallocExtension::Register 2024-09-11T14:36:06.849 INFO:teuthology.orchestra.run.smithi188.stderr:----------------------------------------------------- 2024-09-11T14:36:06.849 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-09-11T14:36:06.872 DEBUG:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-11T14:36:06.897 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-11T14:36:06.897 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-11T14:36:06.897 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-11T14:36:06.963 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-11T14:36:06.963 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-11T14:36:07.028 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-11T14:36:07.028 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-11T14:36:07.093 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-11T14:36:07.093 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-11T14:36:07.157 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-11T14:36:07.157 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-11T14:36:07.213 INFO:tasks.ceph:Adding keys to all mons... 2024-09-11T14:36:07.213 DEBUG:teuthology.orchestra.run.smithi188:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-11T14:36:07.275 INFO:teuthology.orchestra.run.smithi188.stdout:[mgr.x] 2024-09-11T14:36:07.275 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQDKquFmfOx8JBAA0tvKOSguNvvDFQIO6x/Mmw== 2024-09-11T14:36:07.275 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.0] 2024-09-11T14:36:07.275 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQDOquFmgiLkBRAAg93jNuRA7ZfyvzXva711Aw== 2024-09-11T14:36:07.275 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.1] 2024-09-11T14:36:07.276 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQDRquFmkCDMGBAAPRvsPmJVBwFxDbnZx2a0Jw== 2024-09-11T14:36:07.276 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.2] 2024-09-11T14:36:07.276 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQDUquFmA5imKxAAyMZ9rX/joXvfPmj/AQeI+Q== 2024-09-11T14:36:07.276 INFO:teuthology.orchestra.run.smithi188.stdout:[client.0] 2024-09-11T14:36:07.276 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQDKquFmMI2dJxAAOlPW5N7nCNTdVAqFEsySvA== 2024-09-11T14:36:07.277 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-11T14:36:07.379 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T14:36:07.440 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T14:36:07.501 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T14:36:07.562 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T14:36:07.627 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-11T14:36:07.627 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-11T14:36:07.653 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T14:36:07.771 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:07.769+0000 7f22b18d0c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:07.771 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:07.769+0000 7f22b18d0c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:07.784 DEBUG:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-11T14:36:07.809 DEBUG:teuthology.orchestra.run.smithi188:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:36:07.864 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-11T14:36:07.864 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T14:36:07.864 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T14:36:07.907 INFO:tasks.ceph.mon.a:Started 2024-09-11T14:36:07.907 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-11T14:36:07.907 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-11T14:36:07.907 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-11T14:36:07.910 INFO:tasks.ceph.mgr.x:Started 2024-09-11T14:36:07.910 DEBUG:tasks.ceph:set 0 configs 2024-09-11T14:36:07.910 INFO:tasks.ceph:Setting crush tunables to default 2024-09-11T14:36:07.910 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd crush tunables default 2024-09-11T14:36:07.989 INFO:tasks.ceph.mon.a.smithi188.stderr:2024-09-11T14:36:07.988+0000 7f53ad0d5c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:07.990 INFO:tasks.ceph.mon.a.smithi188.stderr:2024-09-11T14:36:07.988+0000 7f53ad0d5c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:07.990 INFO:tasks.ceph.mon.a.smithi188.stderr:2024-09-11T14:36:07.988+0000 7f53ad0d5c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:07.990 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-09-11T14:36:07.989+0000 7f9a65fc3000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:08.020 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:08.013+0000 7f005b591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:08.025 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:08.024+0000 7f005b591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:08.095 INFO:teuthology.orchestra.run.smithi188.stderr:adjusted tunables profile to default 2024-09-11T14:36:08.106 INFO:tasks.ceph:check_enable_crimson: True 2024-09-11T14:36:08.107 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-09-11T14:36:08.107 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-09-11T14:36:08.196 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-09-11T14:36:08.194+0000 7f9a65fc3000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:08.196 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-09-11T14:36:08.194+0000 7f9a65fc3000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:08.213 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:08.207+0000 7f3ae9791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:08.217 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:08.216+0000 7f3ae9791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:08.299 INFO:tasks.ceph.mon.a.smithi188.stderr:2024-09-11T14:36:08.297+0000 7f53a3df4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-09-11T14:36:08.299 INFO:tasks.ceph.mon.a.smithi188.stderr:Please be aware that this feature is experimental, untested, 2024-09-11T14:36:08.299 INFO:tasks.ceph.mon.a.smithi188.stderr:unsupported, and may result in data corruption, data loss, 2024-09-11T14:36:08.299 INFO:tasks.ceph.mon.a.smithi188.stderr:and/or irreparable damage to your cluster. Do not use 2024-09-11T14:36:08.299 INFO:tasks.ceph.mon.a.smithi188.stderr:feature with important data. 2024-09-11T14:36:08.299 INFO:tasks.ceph.mon.a.smithi188.stderr: 2024-09-11T14:36:09.108 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-11T14:36:09.108 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-11T14:36:09.108 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-11T14:36:09.132 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-11T14:36:09.132 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-11T14:36:09.197 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-11T14:36:09.197 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-11T14:36:09.262 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new d19dbbd7-da6f-45c6-8e59-2023cb4b8262 0 2024-09-11T14:36:09.411 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:09.403+0000 7fb5e804f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:09.414 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:09.413+0000 7fb5e804f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:09.493 INFO:teuthology.orchestra.run.smithi188.stdout:0 2024-09-11T14:36:09.505 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new e977da2d-0945-4956-8491-322cd072ca16 1 2024-09-11T14:36:09.612 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:09.605+0000 7fbd6cd91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:09.616 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:09.615+0000 7fbd6cd91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:09.696 INFO:teuthology.orchestra.run.smithi188.stdout:1 2024-09-11T14:36:09.708 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new 8bb79e5a-9a64-4df7-9335-0cf03ab348e5 2 2024-09-11T14:36:09.815 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:09.808+0000 7f72a6afd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:09.819 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:09.818+0000 7f72a6afd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:09.899 INFO:teuthology.orchestra.run.smithi188.stdout:2 2024-09-11T14:36:09.911 DEBUG:teuthology.orchestra.run.smithi188:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-09-11T14:36:09.941 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-11T14:36:09.942 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-09-11T14:36:09.984 INFO:tasks.ceph.osd.0:Started 2024-09-11T14:36:09.984 DEBUG:teuthology.orchestra.run.smithi188:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-09-11T14:36:10.012 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-11T14:36:10.013 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-11T14:36:10.056 INFO:tasks.ceph.osd.1:Started 2024-09-11T14:36:10.056 DEBUG:teuthology.orchestra.run.smithi188:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-09-11T14:36:10.089 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-11T14:36:10.089 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-09-11T14:36:10.132 INFO:tasks.ceph.osd.2:Started 2024-09-11T14:36:10.133 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T14:36:10.261 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:10.255+0000 7feb06351640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:10.265 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:10.264+0000 7feb06351640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:10.334 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:36:10.334 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":6,"fsid":"1c448697-f86d-4ae7-8a96-5392aad74a40","created":"2024-09-11T14:36:08.010044+0000","modified":"2024-09-11T14:36:09.898291+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T14:36:09.898291+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"d19dbbd7-da6f-45c6-8e59-2023cb4b8262","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e977da2d-0945-4956-8491-322cd072ca16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8bb79e5a-9a64-4df7-9335-0cf03ab348e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:36:10.349 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-11T14:36:10.350 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-11T14:36:10.652 DEBUG:teuthology.orchestra.run.smithi188:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T14:36:10.759 INFO:teuthology.misc.health.smithi188.stderr:2024-09-11T14:36:10.753+0000 7fedac888640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:10.763 INFO:teuthology.misc.health.smithi188.stderr:2024-09-11T14:36:10.762+0000 7fedac888640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:10.831 INFO:teuthology.misc.health.smithi188.stdout: 2024-09-11T14:36:10.832 INFO:teuthology.misc.health.smithi188.stdout:{"epoch":6,"fsid":"1c448697-f86d-4ae7-8a96-5392aad74a40","created":"2024-09-11T14:36:08.010044+0000","modified":"2024-09-11T14:36:09.898291+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T14:36:09.898291+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"d19dbbd7-da6f-45c6-8e59-2023cb4b8262","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e977da2d-0945-4956-8491-322cd072ca16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8bb79e5a-9a64-4df7-9335-0cf03ab348e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:36:10.842 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-11T14:36:13.843 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-09-11T14:36:13.841+0000 7f9a03257640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-11T14:36:17.145 DEBUG:teuthology.orchestra.run.smithi188:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T14:36:17.252 INFO:teuthology.misc.health.smithi188.stderr:2024-09-11T14:36:17.246+0000 7fbe336fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:17.256 INFO:teuthology.misc.health.smithi188.stderr:2024-09-11T14:36:17.255+0000 7fbe336fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:17.453 INFO:teuthology.misc.health.smithi188.stdout: 2024-09-11T14:36:17.454 INFO:teuthology.misc.health.smithi188.stdout:{"epoch":10,"fsid":"1c448697-f86d-4ae7-8a96-5392aad74a40","created":"2024-09-11T14:36:08.010044+0000","modified":"2024-09-11T14:36:16.915148+0000","last_up_change":"2024-09-11T14:36:15.903577+0000","last_in_change":"2024-09-11T14:36:09.898291+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T14:36:16.855699+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d19dbbd7-da6f-45c6-8e59-2023cb4b8262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3554493692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6807","nonce":3554493692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":3554493692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6811","nonce":3554493692}]},"public_addr":"172.21.15.188:6806/3554493692","cluster_addr":"172.21.15.188:6807/3554493692","heartbeat_back_addr":"172.21.15.188:6812/3554493692","heartbeat_front_addr":"172.21.15.188:6811/3554493692","state":["exists","up"]},{"osd":1,"uuid":"e977da2d-0945-4956-8491-322cd072ca16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6803","nonce":1156335168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":1156335168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":1156335168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6805","nonce":1156335168}]},"public_addr":"172.21.15.188:6803/1156335168","cluster_addr":"172.21.15.188:6804/1156335168","heartbeat_back_addr":"172.21.15.188:6802/1156335168","heartbeat_front_addr":"172.21.15.188:6805/1156335168","state":["exists","up"]},{"osd":2,"uuid":"8bb79e5a-9a64-4df7-9335-0cf03ab348e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":847045712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6809","nonce":847045712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":847045712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":847045712}]},"public_addr":"172.21.15.188:6810/847045712","cluster_addr":"172.21.15.188:6809/847045712","heartbeat_back_addr":"172.21.15.188:6813/847045712","heartbeat_front_addr":"172.21.15.188:6808/847045712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:36:17.468 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-11T14:36:17.468 INFO:tasks.ceph:Creating RBD pool 2024-09-11T14:36:17.468 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-11T14:36:17.584 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:17.575+0000 7f50b0191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:17.588 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:17.587+0000 7f50b0191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:17.923 INFO:teuthology.orchestra.run.smithi188.stderr:pool 'rbd' created 2024-09-11T14:36:17.940 DEBUG:teuthology.orchestra.run.smithi188:> rbd --cluster ceph pool init rbd 2024-09-11T14:36:18.038 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:18.035+0000 7eff21e5dac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:18.038 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:18.035+0000 7eff21e5dac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:18.043 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:18.041+0000 7eff21e5dac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:21.123 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-11T14:36:21.124 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-11T14:36:21.124 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-11T14:36:21.233 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:21.225+0000 7fc4d1288640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:21.236 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:21.234+0000 7fc4d1288640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:21.422 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:36:21.436 INFO:teuthology.orchestra.run.smithi188.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2024-09-11T14:36:21.436 INFO:tasks.ceph_manager:config epoch is 1 2024-09-11T14:36:21.436 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-11T14:36:21.436 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-11T14:36:21.436 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-11T14:36:21.543 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:21.537+0000 7fb591d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:21.547 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:21.545+0000 7fb591d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:21.778 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:36:21.789 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3250019002},{"type":"v1","addr":"172.21.15.188:6801","nonce":3250019002}]},"active_addr":"172.21.15.188:6801/3250019002","active_change":"2024-09-11T14:36:12.826220+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.188:0","nonce":2529180381}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.188:0","nonce":994306239}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.188:0","nonce":3733676532}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.188:0","nonce":2726046515}]}]} 2024-09-11T14:36:21.792 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-11T14:36:21.792 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-11T14:36:21.792 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T14:36:21.900 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:21.893+0000 7fca248ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:21.903 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:21.901+0000 7fca248ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:22.090 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:36:22.090 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":14,"fsid":"1c448697-f86d-4ae7-8a96-5392aad74a40","created":"2024-09-11T14:36:08.010044+0000","modified":"2024-09-11T14:36:21.043253+0000","last_up_change":"2024-09-11T14:36:15.903577+0000","last_in_change":"2024-09-11T14:36:09.898291+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T14:36:16.855699+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T14:36:17.783600+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d19dbbd7-da6f-45c6-8e59-2023cb4b8262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3554493692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6807","nonce":3554493692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":3554493692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6811","nonce":3554493692}]},"public_addr":"172.21.15.188:6806/3554493692","cluster_addr":"172.21.15.188:6807/3554493692","heartbeat_back_addr":"172.21.15.188:6812/3554493692","heartbeat_front_addr":"172.21.15.188:6811/3554493692","state":["exists","up"]},{"osd":1,"uuid":"e977da2d-0945-4956-8491-322cd072ca16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6803","nonce":1156335168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":1156335168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":1156335168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6805","nonce":1156335168}]},"public_addr":"172.21.15.188:6803/1156335168","cluster_addr":"172.21.15.188:6804/1156335168","heartbeat_back_addr":"172.21.15.188:6802/1156335168","heartbeat_front_addr":"172.21.15.188:6805/1156335168","state":["exists","up"]},{"osd":2,"uuid":"8bb79e5a-9a64-4df7-9335-0cf03ab348e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":847045712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6809","nonce":847045712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":847045712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":847045712}]},"public_addr":"172.21.15.188:6810/847045712","cluster_addr":"172.21.15.188:6809/847045712","heartbeat_back_addr":"172.21.15.188:6813/847045712","heartbeat_front_addr":"172.21.15.188:6808/847045712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:36:22.103 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-11T14:36:22.104 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T14:36:22.211 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:22.206+0000 7fbdcb4e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:22.214 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:22.212+0000 7fbdcb4e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:22.401 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:36:22.401 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":14,"fsid":"1c448697-f86d-4ae7-8a96-5392aad74a40","created":"2024-09-11T14:36:08.010044+0000","modified":"2024-09-11T14:36:21.043253+0000","last_up_change":"2024-09-11T14:36:15.903577+0000","last_in_change":"2024-09-11T14:36:09.898291+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T14:36:16.855699+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T14:36:17.783600+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d19dbbd7-da6f-45c6-8e59-2023cb4b8262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3554493692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6807","nonce":3554493692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":3554493692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6811","nonce":3554493692}]},"public_addr":"172.21.15.188:6806/3554493692","cluster_addr":"172.21.15.188:6807/3554493692","heartbeat_back_addr":"172.21.15.188:6812/3554493692","heartbeat_front_addr":"172.21.15.188:6811/3554493692","state":["exists","up"]},{"osd":1,"uuid":"e977da2d-0945-4956-8491-322cd072ca16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6803","nonce":1156335168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":1156335168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":1156335168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6805","nonce":1156335168}]},"public_addr":"172.21.15.188:6803/1156335168","cluster_addr":"172.21.15.188:6804/1156335168","heartbeat_back_addr":"172.21.15.188:6802/1156335168","heartbeat_front_addr":"172.21.15.188:6805/1156335168","state":["exists","up"]},{"osd":2,"uuid":"8bb79e5a-9a64-4df7-9335-0cf03ab348e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":847045712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6809","nonce":847045712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":847045712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":847045712}]},"public_addr":"172.21.15.188:6810/847045712","cluster_addr":"172.21.15.188:6809/847045712","heartbeat_back_addr":"172.21.15.188:6813/847045712","heartbeat_front_addr":"172.21.15.188:6808/847045712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:36:22.415 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-11T14:36:22.415 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-11T14:36:22.416 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-11T14:36:22.532 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:22.526+0000 7ff37f951640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:22.532 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:22.526+0000 7f220ef48640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:22.533 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:22.526+0000 7f9a2d951640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:22.536 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:22.534+0000 7f220ef48640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:22.536 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:22.534+0000 7f9a2d951640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:22.537 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:22.534+0000 7ff37f951640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:22.567 INFO:teuthology.orchestra.run.smithi188.stdout:34359738375 2024-09-11T14:36:22.567 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T14:36:22.569 INFO:teuthology.orchestra.run.smithi188.stdout:34359738375 2024-09-11T14:36:22.569 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T14:36:22.570 INFO:teuthology.orchestra.run.smithi188.stdout:38654705670 2024-09-11T14:36:22.570 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T14:36:22.678 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:22.671+0000 7fa03b991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:22.683 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:22.681+0000 7fa03b991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:22.723 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:22.716+0000 7fabbf951640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:22.723 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:22.716+0000 7f1fc2b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:22.727 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:22.725+0000 7fabbf951640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:22.727 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:22.725+0000 7f1fc2b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:22.900 INFO:teuthology.orchestra.run.smithi188.stdout:34359738373 2024-09-11T14:36:22.914 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738373 for osd.1 2024-09-11T14:36:22.922 INFO:teuthology.orchestra.run.smithi188.stdout:38654705668 2024-09-11T14:36:22.935 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705668 for osd.2 2024-09-11T14:36:22.945 INFO:teuthology.orchestra.run.smithi188.stdout:34359738372 2024-09-11T14:36:22.958 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738372 for osd.0 2024-09-11T14:36:23.915 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T14:36:23.936 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T14:36:23.959 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T14:36:24.046 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:24.036+0000 7fe8974c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:24.050 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:24.049+0000 7fe8974c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:24.082 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:24.075+0000 7f8519551640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:24.082 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:24.076+0000 7faa44b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:24.086 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:24.085+0000 7f8519551640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:24.086 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:24.085+0000 7faa44b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:24.245 INFO:teuthology.orchestra.run.smithi188.stdout:34359738375 2024-09-11T14:36:24.258 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738375 for osd.1 2024-09-11T14:36:24.259 DEBUG:teuthology.parallel:result is None 2024-09-11T14:36:24.281 INFO:teuthology.orchestra.run.smithi188.stdout:38654705670 2024-09-11T14:36:24.284 INFO:teuthology.orchestra.run.smithi188.stdout:34359738375 2024-09-11T14:36:24.296 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705670 for osd.2 2024-09-11T14:36:24.296 DEBUG:teuthology.parallel:result is None 2024-09-11T14:36:24.298 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738375 for osd.0 2024-09-11T14:36:24.298 DEBUG:teuthology.parallel:result is None 2024-09-11T14:36:24.298 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T14:36:24.298 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:36:24.446 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:24.441+0000 7fac05951640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:24.450 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:24.448+0000 7fac05951640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:24.633 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:36:24.633 INFO:teuthology.orchestra.run.smithi188.stderr:dumped all 2024-09-11T14:36:24.645 INFO:teuthology.orchestra.run.smithi188.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-09-11T14:36:22.832885+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3348,"kb_used_data":3348,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889836,"statfs":{"total":160658620416,"available":160655192064,"internally_reserved":0,"allocated":3428352,"data_stored":3428352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.850170"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:36:21.080316+0000","last_change":"2024-09-11T14:36:21.090039+0000","last_active":"2024-09-11T14:36:21.080316+0000","last_peered":"2024-09-11T14:36:21.080316+0000","last_clean":"2024-09-11T14:36:21.080316+0000","last_became_active":"2024-09-11T14:36:19.161959+0000","last_became_peered":"2024-09-11T14:36:19.161959+0000","last_unstale":"2024-09-11T14:36:21.080316+0000","last_undegraded":"2024-09-11T14:36:21.080316+0000","last_fullsized":"2024-09-11T14:36:21.080316+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:36:21.087167+0000","last_change":"2024-09-11T14:36:21.093532+0000","last_active":"2024-09-11T14:36:21.087167+0000","last_peered":"2024-09-11T14:36:21.087167+0000","last_clean":"2024-09-11T14:36:21.087167+0000","last_became_active":"2024-09-11T14:36:19.246709+0000","last_became_peered":"2024-09-11T14:36:19.246709+0000","last_unstale":"2024-09-11T14:36:21.087167+0000","last_undegraded":"2024-09-11T14:36:21.087167+0000","last_fullsized":"2024-09-11T14:36:21.087167+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:36:21.079046+0000","last_change":"2024-09-11T14:36:21.089054+0000","last_active":"2024-09-11T14:36:21.079046+0000","last_peered":"2024-09-11T14:36:21.079046+0000","last_clean":"2024-09-11T14:36:21.079046+0000","last_became_active":"2024-09-11T14:36:19.144086+0000","last_became_peered":"2024-09-11T14:36:19.144086+0000","last_unstale":"2024-09-11T14:36:21.079046+0000","last_undegraded":"2024-09-11T14:36:21.079046+0000","last_fullsized":"2024-09-11T14:36:21.079046+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:36:21.076918+0000","last_change":"2024-09-11T14:36:21.089791+0000","last_active":"2024-09-11T14:36:21.076918+0000","last_peered":"2024-09-11T14:36:21.076918+0000","last_clean":"2024-09-11T14:36:21.076918+0000","last_became_active":"2024-09-11T14:36:19.177779+0000","last_became_peered":"2024-09-11T14:36:19.177779+0000","last_unstale":"2024-09-11T14:36:21.076918+0000","last_undegraded":"2024-09-11T14:36:21.076918+0000","last_fullsized":"2024-09-11T14:36:21.076918+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:36:21.096241+0000","last_change":"2024-09-11T14:36:21.096241+0000","last_active":"2024-09-11T14:36:21.096241+0000","last_peered":"2024-09-11T14:36:21.096241+0000","last_clean":"2024-09-11T14:36:21.096241+0000","last_became_active":"2024-09-11T14:36:19.212471+0000","last_became_peered":"2024-09-11T14:36:19.212471+0000","last_unstale":"2024-09-11T14:36:21.096241+0000","last_undegraded":"2024-09-11T14:36:21.096241+0000","last_fullsized":"2024-09-11T14:36:21.096241+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:36:21.090435+0000","last_change":"2024-09-11T14:36:21.094831+0000","last_active":"2024-09-11T14:36:21.090435+0000","last_peered":"2024-09-11T14:36:21.090435+0000","last_clean":"2024-09-11T14:36:21.090435+0000","last_became_active":"2024-09-11T14:36:19.234683+0000","last_became_peered":"2024-09-11T14:36:19.234683+0000","last_unstale":"2024-09-11T14:36:21.090435+0000","last_undegraded":"2024-09-11T14:36:21.090435+0000","last_fullsized":"2024-09-11T14:36:21.090435+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:36:21.083663+0000","last_change":"2024-09-11T14:36:21.088624+0000","last_active":"2024-09-11T14:36:21.083663+0000","last_peered":"2024-09-11T14:36:21.083663+0000","last_clean":"2024-09-11T14:36:21.083663+0000","last_became_active":"2024-09-11T14:36:19.162581+0000","last_became_peered":"2024-09-11T14:36:19.162581+0000","last_unstale":"2024-09-11T14:36:21.083663+0000","last_undegraded":"2024-09-11T14:36:21.083663+0000","last_fullsized":"2024-09-11T14:36:21.083663+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:36:21.087859+0000","last_change":"2024-09-11T14:36:21.094074+0000","last_active":"2024-09-11T14:36:21.087859+0000","last_peered":"2024-09-11T14:36:21.087859+0000","last_clean":"2024-09-11T14:36:21.087859+0000","last_became_active":"2024-09-11T14:36:19.167639+0000","last_became_peered":"2024-09-11T14:36:19.167639+0000","last_unstale":"2024-09-11T14:36:21.087859+0000","last_undegraded":"2024-09-11T14:36:21.087859+0000","last_fullsized":"2024-09-11T14:36:21.087859+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":84,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:36:21.079553+0000","last_change":"2024-09-11T14:36:18.038990+0000","last_active":"2024-09-11T14:36:21.079553+0000","last_peered":"2024-09-11T14:36:21.079553+0000","last_clean":"2024-09-11T14:36:21.079553+0000","last_became_active":"2024-09-11T14:36:17.997929+0000","last_became_peered":"2024-09-11T14:36:17.997929+0000","last_unstale":"2024-09-11T14:36:21.079553+0000","last_undegraded":"2024-09-11T14:36:21.079553+0000","last_fullsized":"2024-09-11T14:36:21.079553+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:16.915148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:16.915148+0000","last_clean_scrub_stamp":"2024-09-11T14:36:16.915148+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655192064,"internally_reserved":0,"allocated":3428352,"data_stored":3428352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":428,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297300,"statfs":{"total":53552873472,"available":53552435200,"internally_reserved":0,"allocated":438272,"data_stored":438272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1588,"kb_used_data":1588,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296140,"statfs":{"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1332,"kb_used_data":1332,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296396,"statfs":{"total":53552873472,"available":53551509504,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551509504,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552435200,"internally_reserved":0,"allocated":438272,"data_stored":438272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:36:24.646 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:36:24.754 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:24.748+0000 7fc3b18e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:24.757 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:24.756+0000 7fc3b18e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:24.938 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:36:24.939 INFO:teuthology.orchestra.run.smithi188.stderr:dumped all 2024-09-11T14:36:24.951 INFO:teuthology.orchestra.run.smithi188.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-09-11T14:36:24.833237+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3348,"kb_used_data":3348,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889836,"statfs":{"total":160658620416,"available":160655192064,"internally_reserved":0,"allocated":3428352,"data_stored":3428352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.850522"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:36:21.080316+0000","last_change":"2024-09-11T14:36:21.090039+0000","last_active":"2024-09-11T14:36:21.080316+0000","last_peered":"2024-09-11T14:36:21.080316+0000","last_clean":"2024-09-11T14:36:21.080316+0000","last_became_active":"2024-09-11T14:36:19.161959+0000","last_became_peered":"2024-09-11T14:36:19.161959+0000","last_unstale":"2024-09-11T14:36:21.080316+0000","last_undegraded":"2024-09-11T14:36:21.080316+0000","last_fullsized":"2024-09-11T14:36:21.080316+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:36:21.087167+0000","last_change":"2024-09-11T14:36:21.093532+0000","last_active":"2024-09-11T14:36:21.087167+0000","last_peered":"2024-09-11T14:36:21.087167+0000","last_clean":"2024-09-11T14:36:21.087167+0000","last_became_active":"2024-09-11T14:36:19.246709+0000","last_became_peered":"2024-09-11T14:36:19.246709+0000","last_unstale":"2024-09-11T14:36:21.087167+0000","last_undegraded":"2024-09-11T14:36:21.087167+0000","last_fullsized":"2024-09-11T14:36:21.087167+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:36:21.079046+0000","last_change":"2024-09-11T14:36:21.089054+0000","last_active":"2024-09-11T14:36:21.079046+0000","last_peered":"2024-09-11T14:36:21.079046+0000","last_clean":"2024-09-11T14:36:21.079046+0000","last_became_active":"2024-09-11T14:36:19.144086+0000","last_became_peered":"2024-09-11T14:36:19.144086+0000","last_unstale":"2024-09-11T14:36:21.079046+0000","last_undegraded":"2024-09-11T14:36:21.079046+0000","last_fullsized":"2024-09-11T14:36:21.079046+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:36:21.076918+0000","last_change":"2024-09-11T14:36:21.089791+0000","last_active":"2024-09-11T14:36:21.076918+0000","last_peered":"2024-09-11T14:36:21.076918+0000","last_clean":"2024-09-11T14:36:21.076918+0000","last_became_active":"2024-09-11T14:36:19.177779+0000","last_became_peered":"2024-09-11T14:36:19.177779+0000","last_unstale":"2024-09-11T14:36:21.076918+0000","last_undegraded":"2024-09-11T14:36:21.076918+0000","last_fullsized":"2024-09-11T14:36:21.076918+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:36:21.096241+0000","last_change":"2024-09-11T14:36:21.096241+0000","last_active":"2024-09-11T14:36:21.096241+0000","last_peered":"2024-09-11T14:36:21.096241+0000","last_clean":"2024-09-11T14:36:21.096241+0000","last_became_active":"2024-09-11T14:36:19.212471+0000","last_became_peered":"2024-09-11T14:36:19.212471+0000","last_unstale":"2024-09-11T14:36:21.096241+0000","last_undegraded":"2024-09-11T14:36:21.096241+0000","last_fullsized":"2024-09-11T14:36:21.096241+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:36:21.090435+0000","last_change":"2024-09-11T14:36:21.094831+0000","last_active":"2024-09-11T14:36:21.090435+0000","last_peered":"2024-09-11T14:36:21.090435+0000","last_clean":"2024-09-11T14:36:21.090435+0000","last_became_active":"2024-09-11T14:36:19.234683+0000","last_became_peered":"2024-09-11T14:36:19.234683+0000","last_unstale":"2024-09-11T14:36:21.090435+0000","last_undegraded":"2024-09-11T14:36:21.090435+0000","last_fullsized":"2024-09-11T14:36:21.090435+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:36:21.083663+0000","last_change":"2024-09-11T14:36:21.088624+0000","last_active":"2024-09-11T14:36:21.083663+0000","last_peered":"2024-09-11T14:36:21.083663+0000","last_clean":"2024-09-11T14:36:21.083663+0000","last_became_active":"2024-09-11T14:36:19.162581+0000","last_became_peered":"2024-09-11T14:36:19.162581+0000","last_unstale":"2024-09-11T14:36:21.083663+0000","last_undegraded":"2024-09-11T14:36:21.083663+0000","last_fullsized":"2024-09-11T14:36:21.083663+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:36:21.087859+0000","last_change":"2024-09-11T14:36:21.094074+0000","last_active":"2024-09-11T14:36:21.087859+0000","last_peered":"2024-09-11T14:36:21.087859+0000","last_clean":"2024-09-11T14:36:21.087859+0000","last_became_active":"2024-09-11T14:36:19.167639+0000","last_became_peered":"2024-09-11T14:36:19.167639+0000","last_unstale":"2024-09-11T14:36:21.087859+0000","last_undegraded":"2024-09-11T14:36:21.087859+0000","last_fullsized":"2024-09-11T14:36:21.087859+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":84,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:36:21.079553+0000","last_change":"2024-09-11T14:36:18.038990+0000","last_active":"2024-09-11T14:36:21.079553+0000","last_peered":"2024-09-11T14:36:21.079553+0000","last_clean":"2024-09-11T14:36:21.079553+0000","last_became_active":"2024-09-11T14:36:17.997929+0000","last_became_peered":"2024-09-11T14:36:17.997929+0000","last_unstale":"2024-09-11T14:36:21.079553+0000","last_undegraded":"2024-09-11T14:36:21.079553+0000","last_fullsized":"2024-09-11T14:36:21.079553+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:16.915148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:16.915148+0000","last_clean_scrub_stamp":"2024-09-11T14:36:16.915148+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655192064,"internally_reserved":0,"allocated":3428352,"data_stored":3428352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":428,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297300,"statfs":{"total":53552873472,"available":53552435200,"internally_reserved":0,"allocated":438272,"data_stored":438272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1588,"kb_used_data":1588,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296140,"statfs":{"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1332,"kb_used_data":1332,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296396,"statfs":{"total":53552873472,"available":53551509504,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551509504,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552435200,"internally_reserved":0,"allocated":438272,"data_stored":438272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:36:24.951 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T14:36:24.952 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-11T14:36:24.952 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-11T14:36:24.952 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-11T14:36:25.059 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:25.055+0000 7ffb1ae7d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:25.064 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:36:25.063+0000 7ffb1ae7d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:36:25.284 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:36:25.284 INFO:teuthology.orchestra.run.smithi188.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-11T14:36:25.297 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-11T14:36:25.297 INFO:teuthology.run_tasks:Running task workunit... 2024-09-11T14:36:25.310 INFO:tasks.workunit:Pulling workunits from ref dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 2024-09-11T14:36:25.310 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-11T14:36:25.311 DEBUG:teuthology.orchestra.run.smithi188:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-11T14:36:25.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T14:36:25.326 INFO:teuthology.orchestra.run.smithi188.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-11T14:36:25.326 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-11T14:36:25.381 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-11T14:36:25.381 DEBUG:teuthology.orchestra.run.smithi188:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-11T14:36:25.436 INFO:tasks.workunit:timeout=3h 2024-09-11T14:36:25.436 INFO:tasks.workunit:cleanup=True 2024-09-11T14:36:25.437 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/matan-b/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 2024-09-11T14:36:25.492 INFO:tasks.workunit.client.0.smithi188.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-11T14:37:09.393 INFO:tasks.workunit.client.0.smithi188.stderr:Note: switching to 'dbebf1a3db6db7b5579464ee715edcf1b3e9cce2'. 2024-09-11T14:37:09.393 INFO:tasks.workunit.client.0.smithi188.stderr: 2024-09-11T14:37:09.393 INFO:tasks.workunit.client.0.smithi188.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-11T14:37:09.393 INFO:tasks.workunit.client.0.smithi188.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-11T14:37:09.393 INFO:tasks.workunit.client.0.smithi188.stderr:state without impacting any branches by switching back to a branch. 2024-09-11T14:37:09.393 INFO:tasks.workunit.client.0.smithi188.stderr: 2024-09-11T14:37:09.393 INFO:tasks.workunit.client.0.smithi188.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-11T14:37:09.394 INFO:tasks.workunit.client.0.smithi188.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-11T14:37:09.394 INFO:tasks.workunit.client.0.smithi188.stderr: 2024-09-11T14:37:09.394 INFO:tasks.workunit.client.0.smithi188.stderr: git switch -c 2024-09-11T14:37:09.394 INFO:tasks.workunit.client.0.smithi188.stderr: 2024-09-11T14:37:09.394 INFO:tasks.workunit.client.0.smithi188.stderr:Or undo this operation with: 2024-09-11T14:37:09.394 INFO:tasks.workunit.client.0.smithi188.stderr: 2024-09-11T14:37:09.394 INFO:tasks.workunit.client.0.smithi188.stderr: git switch - 2024-09-11T14:37:09.394 INFO:tasks.workunit.client.0.smithi188.stderr: 2024-09-11T14:37:09.394 INFO:tasks.workunit.client.0.smithi188.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-11T14:37:09.394 INFO:tasks.workunit.client.0.smithi188.stderr: 2024-09-11T14:37:09.394 INFO:tasks.workunit.client.0.smithi188.stderr:HEAD is now at dbebf1a3db6 qa/config/seastore: enable all subsystem debug 2024-09-11T14:37:09.397 DEBUG:teuthology.orchestra.run.smithi188:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-11T14:37:09.461 INFO:tasks.workunit.client.0.smithi188.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-11T14:37:09.463 INFO:tasks.workunit.client.0.smithi188.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-11T14:37:09.463 INFO:tasks.workunit.client.0.smithi188.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-11T14:37:10.079 INFO:tasks.workunit.client.0.smithi188.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-11T14:37:10.148 INFO:tasks.workunit.client.0.smithi188.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-11T14:37:10.189 INFO:tasks.workunit.client.0.smithi188.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-11T14:37:10.190 INFO:tasks.workunit.client.0.smithi188.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-11T14:37:10.190 INFO:tasks.workunit.client.0.smithi188.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-11T14:37:10.229 INFO:tasks.workunit.client.0.smithi188.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-11T14:37:10.233 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-11T14:37:10.233 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-11T14:37:10.289 INFO:tasks.workunit:Running workunits matching rbd/test_librbd_python.sh on client.0... 2024-09-11T14:37:10.290 INFO:tasks.workunit:Running workunit rbd/test_librbd_python.sh... 2024-09-11T14:37:10.290 DEBUG:teuthology.orchestra.run.smithi188:workunit test rbd/test_librbd_python.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh -m 'not skip_if_crimson' 2024-09-11T14:37:10.349 INFO:tasks.workunit.client.0.smithi188.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh 2024-09-11T14:37:10.349 INFO:tasks.workunit.client.0.smithi188.stderr:+ relpath=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/pybind 2024-09-11T14:37:10.349 INFO:tasks.workunit.client.0.smithi188.stderr:+ '[' -n '' ']' 2024-09-11T14:37:10.350 INFO:tasks.workunit.client.0.smithi188.stderr:+ python3 -m pytest -v /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/pybind/test_rbd.py -m 'not skip_if_crimson' 2024-09-11T14:37:10.527 INFO:tasks.workunit.client.0.smithi188.stdout:============================= test session starts ============================== 2024-09-11T14:37:10.527 INFO:tasks.workunit.client.0.smithi188.stdout:platform linux -- Python 3.9.19, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3 2024-09-11T14:37:10.527 INFO:tasks.workunit.client.0.smithi188.stdout:cachedir: .pytest_cache 2024-09-11T14:37:10.527 INFO:tasks.workunit.client.0.smithi188.stdout:rootdir: /home/ubuntu/cephtest/clone.client.0/src/test/pybind, configfile: pytest.ini 2024-09-11T14:37:11.062 INFO:tasks.workunit.client.0.smithi188.stdout:collecting ... collected 191 items / 2 deselected / 189 selected 2024-09-11T14:37:11.062 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-09-11T14:37:14.197 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_version PASSED [ 0%] 2024-09-11T14:37:15.265 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_create PASSED [ 1%] 2024-09-11T14:37:24.808 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_create_defaults PASSED [ 1%] 2024-09-11T14:37:26.316 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_context_manager PASSED [ 2%] 2024-09-11T14:37:29.409 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_open_read_only PASSED [ 2%] 2024-09-11T14:37:31.991 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_open_dne PASSED [ 3%] 2024-09-11T14:37:34.583 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_open_readonly_dne PASSED [ 3%] 2024-09-11T14:37:34.584 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_open_by_id SKIPPED [ 4%] 2024-09-11T14:37:35.731 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_aio_open PASSED [ 4%] 2024-09-11T14:37:35.846 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_remove_dne PASSED [ 5%] 2024-09-11T14:37:35.865 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_list_empty PASSED [ 5%] 2024-09-11T14:37:37.092 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_list PASSED [ 6%] 2024-09-11T14:37:38.122 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_remove_with_progress PASSED [ 6%] 2024-09-11T14:37:39.469 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_remove_canceled PASSED [ 7%] 2024-09-11T14:37:40.480 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_remove_with_progress_except PASSED [ 7%] 2024-09-11T14:37:42.150 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_rename PASSED [ 8%] 2024-09-11T14:37:46.351 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_pool_metadata PASSED [ 8%] 2024-09-11T14:37:46.452 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_config_list PASSED [ 9%] 2024-09-11T14:37:46.530 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_pool_config_set_and_get_and_remove PASSED [ 10%] 2024-09-11T14:37:46.883 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_namespaces PASSED [ 10%] 2024-09-11T14:37:46.884 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_pool_stats SKIPPED [ 11%] 2024-09-11T14:37:46.885 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_features_to_string SKIPPED [ 11%] 2024-09-11T14:37:46.886 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_features_from_string SKIPPED [ 12%] 2024-09-11T14:37:48.129 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_update_features SKIPPED [ 12%] 2024-09-11T14:37:49.281 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_create_with_params SKIPPED [ 13%] 2024-09-11T14:37:50.435 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_id SKIPPED [ 13%] 2024-09-11T14:37:51.588 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_block_name_prefix PASSED [ 14%] 2024-09-11T14:37:52.769 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_data_pool_id PASSED [ 14%] 2024-09-11T14:37:53.937 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_create_timestamp PASSED [ 15%] 2024-09-11T14:37:55.069 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_access_timestamp PASSED [ 15%] 2024-09-11T14:37:56.257 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_modify_timestamp PASSED [ 16%] 2024-09-11T14:37:57.667 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_invalidate_cache PASSED [ 16%] 2024-09-11T14:37:58.887 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_stat PASSED [ 17%] 2024-09-11T14:38:00.036 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_flags PASSED [ 17%] 2024-09-11T14:38:01.418 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_image_auto_close PASSED [ 18%] 2024-09-11T14:38:02.600 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_use_after_close PASSED [ 19%] 2024-09-11T14:38:03.988 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_write PASSED [ 19%] 2024-09-11T14:38:05.383 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_write_with_fadvise_flags PASSED [ 20%] 2024-09-11T14:38:06.779 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_write_zeroes PASSED [ 20%] 2024-09-11T14:38:08.178 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_write_zeroes_thick_provision PASSED [ 21%] 2024-09-11T14:38:09.422 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_read PASSED [ 21%] 2024-09-11T14:38:10.604 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_read_with_fadvise_flags PASSED [ 22%] 2024-09-11T14:38:12.364 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_large_write PASSED [ 22%] 2024-09-11T14:38:13.546 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_large_read PASSED [ 23%] 2024-09-11T14:38:14.962 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_write_read PASSED [ 23%] 2024-09-11T14:38:16.207 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_read_bad_offset PASSED [ 24%] 2024-09-11T14:38:17.632 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_resize PASSED [ 24%] 2024-09-11T14:38:18.977 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_resize_allow_shrink_False PASSED [ 25%] 2024-09-11T14:38:24.145 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_size PASSED [ 25%] 2024-09-11T14:38:25.796 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_resize_down PASSED [ 26%] 2024-09-11T14:38:27.583 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_resize_bytes PASSED [ 26%] 2024-09-11T14:38:30.820 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_copy PASSED [ 27%] 2024-09-11T14:38:33.981 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_copy2 PASSED [ 28%] 2024-09-11T14:38:35.143 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_copy3 SKIPPED [ 28%] 2024-09-11T14:38:36.355 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_deep_copy_clone_v1_to_v1 SKIPPED [ 29%] 2024-09-11T14:38:37.530 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_deep_copy_clone_v1_to_v2 SKIPPED [ 29%] 2024-09-11T14:38:38.734 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_deep_copy_clone_v2_to_v1 SKIPPED [ 30%] 2024-09-11T14:38:39.903 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_deep_copy_clone_v2_to_v2 SKIPPED [ 30%] 2024-09-11T14:38:41.167 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_deep_copy_clone_v1_flatten SKIPPED [ 31%] 2024-09-11T14:38:42.377 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_deep_copy_clone_v2_flatten SKIPPED [ 31%] 2024-09-11T14:38:45.271 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_create_snap PASSED [ 32%] 2024-09-11T14:38:48.261 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_create_snap_exists PASSED [ 32%] 2024-09-11T14:38:55.257 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_create_snap_flags PASSED [ 33%] 2024-09-11T14:39:00.242 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_list_snaps PASSED [ 33%] 2024-09-11T14:39:03.246 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_list_snaps_iterator_auto_close PASSED [ 34%] 2024-09-11T14:39:06.324 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_remove_snap PASSED [ 34%] 2024-09-11T14:39:07.601 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_remove_snap_not_found PASSED [ 35%] 2024-09-11T14:39:08.803 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_remove_snap2 SKIPPED [ 35%] 2024-09-11T14:39:11.363 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_remove_snap_by_id PASSED [ 36%] 2024-09-11T14:39:14.349 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_rename_snap PASSED [ 37%] 2024-09-11T14:39:15.564 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_protect_snap SKIPPED [ 37%] 2024-09-11T14:39:18.386 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_snap_exists PASSED [ 38%] 2024-09-11T14:39:21.385 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_snap_timestamp PASSED [ 38%] 2024-09-11T14:39:30.402 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_limit_snaps PASSED [ 39%] 2024-09-11T14:39:31.619 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_remove_with_exclusive_lock SKIPPED [ 39%] 2024-09-11T14:39:34.381 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_remove_with_snap PASSED [ 40%] 2024-09-11T14:39:36.087 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_remove_with_watcher PASSED [ 40%] 2024-09-11T14:39:38.739 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_rollback_to_snap PASSED [ 41%] 2024-09-11T14:39:41.666 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_rollback_to_snap_sparse PASSED [ 41%] 2024-09-11T14:39:46.835 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_rollback_with_resize PASSED [ 42%] 2024-09-11T14:39:49.239 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_set_snap PASSED [ 42%] 2024-09-11T14:39:52.857 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_set_no_snap PASSED [ 43%] 2024-09-11T14:39:55.799 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_set_snap_by_id PASSED [ 43%] 2024-09-11T14:40:02.775 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_snap_get_name PASSED [ 44%] 2024-09-11T14:40:09.299 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_snap_get_id PASSED [ 44%] 2024-09-11T14:40:11.799 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_set_snap_sparse PASSED [ 45%] 2024-09-11T14:52:16.399 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_many_snaps PASSED [ 46%] 2024-09-11T14:52:19.392 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_set_snap_deleted PASSED [ 46%] 2024-09-11T14:52:24.437 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_set_snap_recreated PASSED [ 47%] 2024-09-11T14:52:25.848 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_lock_unlock PASSED [ 47%] 2024-09-11T14:52:28.916 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_list_lockers PASSED [ 48%] 2024-09-11T14:52:34.299 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_diff_iterate PASSED [ 48%] 2024-09-11T14:52:35.598 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_aio_read PASSED [ 49%] 2024-09-11T14:52:37.069 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_aio_write PASSED [ 49%] 2024-09-11T14:52:38.513 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_aio_discard PASSED [ 50%] 2024-09-11T14:52:40.005 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_aio_write_zeroes PASSED [ 50%] 2024-09-11T14:52:41.403 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_aio_flush PASSED [ 51%] 2024-09-11T14:52:55.652 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_metadata PASSED [ 51%] 2024-09-11T14:52:56.965 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_watchers_list PASSED [ 52%] 2024-09-11T14:52:58.646 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_config_list PASSED [ 52%] 2024-09-11T14:53:00.555 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_image_config_set_and_get_and_remove PASSED [ 53%] 2024-09-11T14:53:01.872 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_sparsify PASSED [ 53%] 2024-09-11T14:53:12.045 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_encryption_luks1 PASSED [ 54%] 2024-09-11T14:53:24.361 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_encryption_luks2 PASSED [ 55%] 2024-09-11T14:53:25.672 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImageId::test_read PASSED [ 55%] 2024-09-11T14:53:27.238 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImageId::test_write PASSED [ 56%] 2024-09-11T14:53:28.837 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImageId::test_resize PASSED [ 56%] 2024-09-11T14:53:28.838 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_with_params SKIPPED [ 57%] 2024-09-11T14:53:28.838 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_with_params2 SKIPPED [ 57%] 2024-09-11T14:53:28.839 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_with_params3 SKIPPED [ 58%] 2024-09-11T14:53:28.840 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_stripe_unit_and_count SKIPPED [ 58%] 2024-09-11T14:53:28.841 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_clone_format SKIPPED [ 59%] 2024-09-11T14:53:28.842 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_unprotected SKIPPED [ 59%] 2024-09-11T14:53:28.843 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_unprotect_with_children SKIPPED [ 60%] 2024-09-11T14:53:28.844 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_clone_by_snap_id SKIPPED [ 60%] 2024-09-11T14:53:28.845 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_stat SKIPPED [ 61%] 2024-09-11T14:53:28.846 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_resize_stat SKIPPED [ 61%] 2024-09-11T14:53:28.847 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_resize_io SKIPPED [ 62%] 2024-09-11T14:53:28.848 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_read SKIPPED [ 62%] 2024-09-11T14:53:28.850 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_write SKIPPED [ 63%] 2024-09-11T14:53:28.851 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_list_children SKIPPED [ 64%] 2024-09-11T14:53:28.851 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_flatten_errors SKIPPED [ 64%] 2024-09-11T14:53:28.852 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_flatten_basic SKIPPED [ 65%] 2024-09-11T14:53:28.854 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_flatten_smaller_order SKIPPED [ 65%] 2024-09-11T14:53:28.855 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_flatten_larger_order SKIPPED [ 66%] 2024-09-11T14:53:28.857 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_flatten_drops_cache SKIPPED [ 66%] 2024-09-11T14:53:28.858 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_flatten_multi_level SKIPPED [ 67%] 2024-09-11T14:53:28.859 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_flatten_with_progress SKIPPED [ 67%] 2024-09-11T14:53:28.861 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_resize_flatten_multi_level SKIPPED [ 68%] 2024-09-11T14:53:28.862 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_trash_snapshot SKIPPED [ 68%] 2024-09-11T14:53:28.863 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_encryption_luks1 SKIPPED [ 69%] 2024-09-11T14:53:28.864 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_encryption_luks2 SKIPPED [ 69%] 2024-09-11T14:53:28.865 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_ownership SKIPPED [ 70%] 2024-09-11T14:53:28.866 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_snapshot_leadership SKIPPED [ 70%] 2024-09-11T14:53:28.866 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_read_only_leadership SKIPPED [ 71%] 2024-09-11T14:53:28.867 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_follower_flatten SKIPPED [ 71%] 2024-09-11T14:53:28.868 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_follower_resize SKIPPED [ 72%] 2024-09-11T14:53:28.869 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_follower_snap_create SKIPPED [ 73%] 2024-09-11T14:53:28.871 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_follower_snap_rollback SKIPPED [ 73%] 2024-09-11T14:53:28.872 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_follower_discard SKIPPED [ 74%] 2024-09-11T14:53:28.873 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_follower_write SKIPPED [ 74%] 2024-09-11T14:53:28.874 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_acquire_release_lock SKIPPED [ 75%] 2024-09-11T14:53:30.259 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_uuid PASSED [ 75%] 2024-09-11T14:53:31.713 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_site_name PASSED [ 76%] 2024-09-11T14:53:33.268 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_mirror_peer_bootstrap PASSED [ 76%] 2024-09-11T14:53:34.898 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_mirror_peer PASSED [ 77%] 2024-09-11T14:53:36.314 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_mirror_image SKIPPED [ 77%] 2024-09-11T14:53:37.776 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_mirror_image_status SKIPPED [ 78%] 2024-09-11T14:53:47.961 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_mirror_image_create_snapshot PASSED [ 78%] 2024-09-11T14:53:53.917 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_aio_mirror_image_create_snapshot PASSED [ 79%] 2024-09-11T14:53:55.034 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_move PASSED [ 79%] 2024-09-11T14:53:57.065 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_purge PASSED [ 80%] 2024-09-11T14:53:58.081 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_remove_denied PASSED [ 80%] 2024-09-11T14:53:59.082 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_remove PASSED [ 81%] 2024-09-11T14:54:00.074 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_remove_with_progress PASSED [ 82%] 2024-09-11T14:54:01.145 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_get PASSED [ 82%] 2024-09-11T14:54:03.265 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_list PASSED [ 83%] 2024-09-11T14:54:05.018 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_restore PASSED [ 83%] 2024-09-11T14:54:05.121 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_create_group PASSED [ 84%] 2024-09-11T14:54:05.263 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_rename_group PASSED [ 84%] 2024-09-11T14:54:05.271 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_list_groups_empty PASSED [ 85%] 2024-09-11T14:54:05.382 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_list_groups PASSED [ 85%] 2024-09-11T14:54:05.489 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_list_groups_after_removed PASSED [ 86%] 2024-09-11T14:54:07.000 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_image_add PASSED [ 86%] 2024-09-11T14:54:08.356 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_image_list_empty PASSED [ 87%] 2024-09-11T14:54:09.973 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_image_list PASSED [ 87%] 2024-09-11T14:54:12.212 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_image_list_move_to_trash PASSED [ 88%] 2024-09-11T14:54:13.610 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_get_id PASSED [ 88%] 2024-09-11T14:54:41.373 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_image_many_images PASSED [ 89%] 2024-09-11T14:54:43.199 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_image_remove PASSED [ 89%] 2024-09-11T14:54:47.713 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_get_info PASSED [ 90%] 2024-09-11T14:54:49.221 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_get_info_no_member_images PASSED [ 91%] 2024-09-11T14:54:50.832 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap PASSED [ 91%] 2024-09-11T14:54:52.695 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_flags PASSED [ 92%] 2024-09-11T14:55:50.022 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_list_many PASSED [ 92%] 2024-09-11T14:55:53.698 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_namespace PASSED [ 93%] 2024-09-11T14:55:55.323 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_rename PASSED [ 93%] 2024-09-11T14:55:56.695 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_clone SKIPPED [ 94%] 2024-09-11T14:55:58.049 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_clone_flatten SKIPPED [ 94%] 2024-09-11T14:56:21.190 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_rollback PASSED [ 95%] 2024-09-11T14:56:25.679 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migration PASSED [ 95%] 2024-09-11T14:56:25.680 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migration_clone_v1_to_v1 SKIPPED [ 96%] 2024-09-11T14:56:25.681 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migration_clone_v1_to_v2 SKIPPED [ 96%] 2024-09-11T14:56:25.682 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migration_clone_v2_to_v1 SKIPPED [ 97%] 2024-09-11T14:56:25.683 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migration_clone_v2_to_v2 SKIPPED [ 97%] 2024-09-11T14:56:33.722 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migration_import PASSED [ 98%] 2024-09-11T14:56:37.568 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migration_with_progress PASSED [ 98%] 2024-09-11T14:56:40.957 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migrate_abort PASSED [ 99%] 2024-09-11T14:56:44.366 INFO:tasks.workunit.client.0.smithi188.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migrate_abort_with_progress PASSED [100%] 2024-09-11T14:56:44.366 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-09-11T14:56:44.366 INFO:tasks.workunit.client.0.smithi188.stdout:=============================== warnings summary =============================== 2024-09-11T14:56:44.366 INFO:tasks.workunit.client.0.smithi188.stdout:test_rbd.py::test_remove_with_progress_except 2024-09-11T14:56:44.366 INFO:tasks.workunit.client.0.smithi188.stdout: /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning: Exception ignored in: 'rbd.progress_callback' 2024-09-11T14:56:44.366 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-09-11T14:56:44.366 INFO:tasks.workunit.client.0.smithi188.stdout: Traceback (most recent call last): 2024-09-11T14:56:44.367 INFO:tasks.workunit.client.0.smithi188.stdout: File "/home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_rbd.py", line 427, in progress_cb 2024-09-11T14:56:44.367 INFO:tasks.workunit.client.0.smithi188.stdout: raise Exception() 2024-09-11T14:56:44.367 INFO:tasks.workunit.client.0.smithi188.stdout: Exception 2024-09-11T14:56:44.367 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-09-11T14:56:44.367 INFO:tasks.workunit.client.0.smithi188.stdout: warnings.warn(pytest.PytestUnraisableExceptionWarning(msg)) 2024-09-11T14:56:44.367 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-09-11T14:56:44.367 INFO:tasks.workunit.client.0.smithi188.stdout:-- Docs: https://docs.pytest.org/en/stable/warnings.html 2024-09-11T14:56:44.367 INFO:tasks.workunit.client.0.smithi188.stdout:==== 129 passed, 60 skipped, 2 deselected, 1 warning in 1173.84s (0:19:33) ===== 2024-09-11T14:56:44.418 INFO:tasks.workunit.client.0.smithi188.stderr:+ exit 0 2024-09-11T14:56:44.418 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-11T14:56:44.419 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-11T14:56:44.491 INFO:tasks.workunit:Stopping ["rbd/test_librbd_python.sh -m 'not skip_if_crimson'"] on client.0... 2024-09-11T14:56:44.491 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-11T14:56:44.965 DEBUG:teuthology.parallel:result is None 2024-09-11T14:56:44.965 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-11T14:56:44.993 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-11T14:56:44.993 DEBUG:teuthology.orchestra.run.smithi188:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-11T14:56:45.063 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-11T14:56:45.063 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-11T14:56:45.074 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T14:56:45.074 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:56:45.261 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:56:45.253+0000 7f131cd91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:56:45.265 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:56:45.263+0000 7f131cd91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:56:45.498 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:56:45.498 INFO:teuthology.orchestra.run.smithi188.stderr:dumped all 2024-09-11T14:56:45.516 INFO:teuthology.orchestra.run.smithi188.stdout:{"pg_ready":true,"pg_map":{"version":1208,"stamp":"2024-09-11T14:56:45.082482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":16600,"kb_used_data":16600,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156876584,"statfs":{"total":160658620416,"available":160641622016,"internally_reserved":0,"allocated":16998400,"data_stored":16998400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-164,"num_objects":-12,"num_object_clones":0,"num_object_copies":-24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":-338505,"num_read_kb":-11144,"num_write":-7833,"num_write_kb":-41271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002185"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1767,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-09-11T14:56:31.464779+0000","last_change":"2024-09-11T14:36:21.090039+0000","last_active":"2024-09-11T14:56:31.464779+0000","last_peered":"2024-09-11T14:56:31.464779+0000","last_clean":"2024-09-11T14:56:31.464779+0000","last_became_active":"2024-09-11T14:36:19.161959+0000","last_became_peered":"2024-09-11T14:36:19.161959+0000","last_unstale":"2024-09-11T14:56:31.464779+0000","last_undegraded":"2024-09-11T14:56:31.464779+0000","last_fullsized":"2024-09-11T14:56:31.464779+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1767,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-09-11T14:56:31.470280+0000","last_change":"2024-09-11T14:36:21.093532+0000","last_active":"2024-09-11T14:56:31.470280+0000","last_peered":"2024-09-11T14:56:31.470280+0000","last_clean":"2024-09-11T14:56:31.470280+0000","last_became_active":"2024-09-11T14:36:19.246709+0000","last_became_peered":"2024-09-11T14:36:19.246709+0000","last_unstale":"2024-09-11T14:56:31.470280+0000","last_undegraded":"2024-09-11T14:56:31.470280+0000","last_fullsized":"2024-09-11T14:56:31.470280+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1767,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-09-11T14:56:31.474047+0000","last_change":"2024-09-11T14:36:21.089054+0000","last_active":"2024-09-11T14:56:31.474047+0000","last_peered":"2024-09-11T14:56:31.474047+0000","last_clean":"2024-09-11T14:56:31.474047+0000","last_became_active":"2024-09-11T14:36:19.144086+0000","last_became_peered":"2024-09-11T14:36:19.144086+0000","last_unstale":"2024-09-11T14:56:31.474047+0000","last_undegraded":"2024-09-11T14:56:31.474047+0000","last_fullsized":"2024-09-11T14:56:31.474047+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1767,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-09-11T14:56:31.470261+0000","last_change":"2024-09-11T14:36:21.089791+0000","last_active":"2024-09-11T14:56:31.470261+0000","last_peered":"2024-09-11T14:56:31.470261+0000","last_clean":"2024-09-11T14:56:31.470261+0000","last_became_active":"2024-09-11T14:36:19.177779+0000","last_became_peered":"2024-09-11T14:36:19.177779+0000","last_unstale":"2024-09-11T14:56:31.470261+0000","last_undegraded":"2024-09-11T14:56:31.470261+0000","last_fullsized":"2024-09-11T14:56:31.470261+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":1772,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-11T14:56:44.577410+0000","last_change":"2024-09-11T14:36:21.096241+0000","last_active":"2024-09-11T14:56:44.577410+0000","last_peered":"2024-09-11T14:56:44.577410+0000","last_clean":"2024-09-11T14:56:44.577410+0000","last_became_active":"2024-09-11T14:36:19.212471+0000","last_became_peered":"2024-09-11T14:36:19.212471+0000","last_unstale":"2024-09-11T14:56:44.577410+0000","last_undegraded":"2024-09-11T14:56:44.577410+0000","last_fullsized":"2024-09-11T14:56:44.577410+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1767,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-09-11T14:56:31.519122+0000","last_change":"2024-09-11T14:36:21.094831+0000","last_active":"2024-09-11T14:56:31.519122+0000","last_peered":"2024-09-11T14:56:31.519122+0000","last_clean":"2024-09-11T14:56:31.519122+0000","last_became_active":"2024-09-11T14:36:19.234683+0000","last_became_peered":"2024-09-11T14:36:19.234683+0000","last_unstale":"2024-09-11T14:56:31.519122+0000","last_undegraded":"2024-09-11T14:56:31.519122+0000","last_fullsized":"2024-09-11T14:56:31.519122+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1767,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-09-11T14:56:31.508868+0000","last_change":"2024-09-11T14:36:21.088624+0000","last_active":"2024-09-11T14:56:31.508868+0000","last_peered":"2024-09-11T14:56:31.508868+0000","last_clean":"2024-09-11T14:56:31.508868+0000","last_became_active":"2024-09-11T14:36:19.162581+0000","last_became_peered":"2024-09-11T14:36:19.162581+0000","last_unstale":"2024-09-11T14:56:31.508868+0000","last_undegraded":"2024-09-11T14:56:31.508868+0000","last_fullsized":"2024-09-11T14:56:31.508868+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":1768,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-09-11T14:56:31.481881+0000","last_change":"2024-09-11T14:36:21.094074+0000","last_active":"2024-09-11T14:56:31.481881+0000","last_peered":"2024-09-11T14:56:31.481881+0000","last_clean":"2024-09-11T14:56:31.481881+0000","last_became_active":"2024-09-11T14:36:19.167639+0000","last_became_peered":"2024-09-11T14:36:19.167639+0000","last_unstale":"2024-09-11T14:56:31.481881+0000","last_undegraded":"2024-09-11T14:56:31.481881+0000","last_fullsized":"2024-09-11T14:56:31.481881+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":1836,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-09-11T14:56:31.461315+0000","last_change":"2024-09-11T14:36:18.038990+0000","last_active":"2024-09-11T14:56:31.461315+0000","last_peered":"2024-09-11T14:56:31.461315+0000","last_clean":"2024-09-11T14:56:31.461315+0000","last_became_active":"2024-09-11T14:36:17.997929+0000","last_became_peered":"2024-09-11T14:36:17.997929+0000","last_unstale":"2024-09-11T14:56:31.461315+0000","last_undegraded":"2024-09-11T14:56:31.461315+0000","last_fullsized":"2024-09-11T14:56:31.461315+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:16.915148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:16.915148+0000","last_clean_scrub_stamp":"2024-09-11T14:36:16.915148+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160642633728,"internally_reserved":0,"allocated":15986688,"data_stored":15986688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53545910272,"internally_reserved":0,"allocated":6963200,"data_stored":6963200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4472,"kb_used_data":4472,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293256,"statfs":{"total":53552873472,"available":53548294144,"internally_reserved":0,"allocated":4579328,"data_stored":4579328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739592,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":6860,"kb_used_data":6860,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52290868,"statfs":{"total":53552873472,"available":53545848832,"internally_reserved":0,"allocated":7024640,"data_stored":7024640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739597,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":5268,"kb_used_data":5268,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52292460,"statfs":{"total":53552873472,"available":53547479040,"internally_reserved":0,"allocated":5394432,"data_stored":5394432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53545910272,"internally_reserved":0,"allocated":6963200,"data_stored":6963200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53548228608,"internally_reserved":0,"allocated":4644864,"data_stored":4644864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53545910272,"internally_reserved":0,"allocated":6963200,"data_stored":6963200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53548494848,"internally_reserved":0,"allocated":4378624,"data_stored":4378624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:56:45.517 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:56:45.627 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:56:45.621+0000 7f32ea151640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:56:45.630 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:56:45.628+0000 7f32ea151640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:56:45.813 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:56:45.814 INFO:teuthology.orchestra.run.smithi188.stderr:dumped all 2024-09-11T14:56:45.827 INFO:teuthology.orchestra.run.smithi188.stdout:{"pg_ready":true,"pg_map":{"version":1208,"stamp":"2024-09-11T14:56:45.082482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":16600,"kb_used_data":16600,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156876584,"statfs":{"total":160658620416,"available":160641622016,"internally_reserved":0,"allocated":16998400,"data_stored":16998400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-164,"num_objects":-12,"num_object_clones":0,"num_object_copies":-24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":-338505,"num_read_kb":-11144,"num_write":-7833,"num_write_kb":-41271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002185"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1767,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-09-11T14:56:31.464779+0000","last_change":"2024-09-11T14:36:21.090039+0000","last_active":"2024-09-11T14:56:31.464779+0000","last_peered":"2024-09-11T14:56:31.464779+0000","last_clean":"2024-09-11T14:56:31.464779+0000","last_became_active":"2024-09-11T14:36:19.161959+0000","last_became_peered":"2024-09-11T14:36:19.161959+0000","last_unstale":"2024-09-11T14:56:31.464779+0000","last_undegraded":"2024-09-11T14:56:31.464779+0000","last_fullsized":"2024-09-11T14:56:31.464779+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1767,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-09-11T14:56:31.470280+0000","last_change":"2024-09-11T14:36:21.093532+0000","last_active":"2024-09-11T14:56:31.470280+0000","last_peered":"2024-09-11T14:56:31.470280+0000","last_clean":"2024-09-11T14:56:31.470280+0000","last_became_active":"2024-09-11T14:36:19.246709+0000","last_became_peered":"2024-09-11T14:36:19.246709+0000","last_unstale":"2024-09-11T14:56:31.470280+0000","last_undegraded":"2024-09-11T14:56:31.470280+0000","last_fullsized":"2024-09-11T14:56:31.470280+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1767,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-09-11T14:56:31.474047+0000","last_change":"2024-09-11T14:36:21.089054+0000","last_active":"2024-09-11T14:56:31.474047+0000","last_peered":"2024-09-11T14:56:31.474047+0000","last_clean":"2024-09-11T14:56:31.474047+0000","last_became_active":"2024-09-11T14:36:19.144086+0000","last_became_peered":"2024-09-11T14:36:19.144086+0000","last_unstale":"2024-09-11T14:56:31.474047+0000","last_undegraded":"2024-09-11T14:56:31.474047+0000","last_fullsized":"2024-09-11T14:56:31.474047+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1767,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-09-11T14:56:31.470261+0000","last_change":"2024-09-11T14:36:21.089791+0000","last_active":"2024-09-11T14:56:31.470261+0000","last_peered":"2024-09-11T14:56:31.470261+0000","last_clean":"2024-09-11T14:56:31.470261+0000","last_became_active":"2024-09-11T14:36:19.177779+0000","last_became_peered":"2024-09-11T14:36:19.177779+0000","last_unstale":"2024-09-11T14:56:31.470261+0000","last_undegraded":"2024-09-11T14:56:31.470261+0000","last_fullsized":"2024-09-11T14:56:31.470261+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":1772,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-11T14:56:44.577410+0000","last_change":"2024-09-11T14:36:21.096241+0000","last_active":"2024-09-11T14:56:44.577410+0000","last_peered":"2024-09-11T14:56:44.577410+0000","last_clean":"2024-09-11T14:56:44.577410+0000","last_became_active":"2024-09-11T14:36:19.212471+0000","last_became_peered":"2024-09-11T14:36:19.212471+0000","last_unstale":"2024-09-11T14:56:44.577410+0000","last_undegraded":"2024-09-11T14:56:44.577410+0000","last_fullsized":"2024-09-11T14:56:44.577410+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1767,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-09-11T14:56:31.519122+0000","last_change":"2024-09-11T14:36:21.094831+0000","last_active":"2024-09-11T14:56:31.519122+0000","last_peered":"2024-09-11T14:56:31.519122+0000","last_clean":"2024-09-11T14:56:31.519122+0000","last_became_active":"2024-09-11T14:36:19.234683+0000","last_became_peered":"2024-09-11T14:36:19.234683+0000","last_unstale":"2024-09-11T14:56:31.519122+0000","last_undegraded":"2024-09-11T14:56:31.519122+0000","last_fullsized":"2024-09-11T14:56:31.519122+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1767,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-09-11T14:56:31.508868+0000","last_change":"2024-09-11T14:36:21.088624+0000","last_active":"2024-09-11T14:56:31.508868+0000","last_peered":"2024-09-11T14:56:31.508868+0000","last_clean":"2024-09-11T14:56:31.508868+0000","last_became_active":"2024-09-11T14:36:19.162581+0000","last_became_peered":"2024-09-11T14:36:19.162581+0000","last_unstale":"2024-09-11T14:56:31.508868+0000","last_undegraded":"2024-09-11T14:56:31.508868+0000","last_fullsized":"2024-09-11T14:56:31.508868+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":1768,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-09-11T14:56:31.481881+0000","last_change":"2024-09-11T14:36:21.094074+0000","last_active":"2024-09-11T14:56:31.481881+0000","last_peered":"2024-09-11T14:56:31.481881+0000","last_clean":"2024-09-11T14:56:31.481881+0000","last_became_active":"2024-09-11T14:36:19.167639+0000","last_became_peered":"2024-09-11T14:36:19.167639+0000","last_unstale":"2024-09-11T14:56:31.481881+0000","last_undegraded":"2024-09-11T14:56:31.481881+0000","last_fullsized":"2024-09-11T14:56:31.481881+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":1836,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-09-11T14:56:31.461315+0000","last_change":"2024-09-11T14:36:18.038990+0000","last_active":"2024-09-11T14:56:31.461315+0000","last_peered":"2024-09-11T14:56:31.461315+0000","last_clean":"2024-09-11T14:56:31.461315+0000","last_became_active":"2024-09-11T14:36:17.997929+0000","last_became_peered":"2024-09-11T14:36:17.997929+0000","last_unstale":"2024-09-11T14:56:31.461315+0000","last_undegraded":"2024-09-11T14:56:31.461315+0000","last_fullsized":"2024-09-11T14:56:31.461315+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:16.915148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:16.915148+0000","last_clean_scrub_stamp":"2024-09-11T14:36:16.915148+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160642633728,"internally_reserved":0,"allocated":15986688,"data_stored":15986688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53545910272,"internally_reserved":0,"allocated":6963200,"data_stored":6963200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4472,"kb_used_data":4472,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293256,"statfs":{"total":53552873472,"available":53548294144,"internally_reserved":0,"allocated":4579328,"data_stored":4579328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739592,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":6860,"kb_used_data":6860,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52290868,"statfs":{"total":53552873472,"available":53545848832,"internally_reserved":0,"allocated":7024640,"data_stored":7024640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739597,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":5268,"kb_used_data":5268,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52292460,"statfs":{"total":53552873472,"available":53547479040,"internally_reserved":0,"allocated":5394432,"data_stored":5394432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53545910272,"internally_reserved":0,"allocated":6963200,"data_stored":6963200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53548228608,"internally_reserved":0,"allocated":4644864,"data_stored":4644864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53545910272,"internally_reserved":0,"allocated":6963200,"data_stored":6963200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53548494848,"internally_reserved":0,"allocated":4378624,"data_stored":4378624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:56:45.828 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T14:56:45.829 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:56:45.937 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:56:45.931+0000 7f4250a46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:56:45.940 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:56:45.938+0000 7f4250a46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:56:46.124 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:56:46.124 INFO:teuthology.orchestra.run.smithi188.stderr:dumped all 2024-09-11T14:56:46.137 INFO:teuthology.orchestra.run.smithi188.stdout:{"pg_ready":true,"pg_map":{"version":1208,"stamp":"2024-09-11T14:56:45.082482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":16600,"kb_used_data":16600,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156876584,"statfs":{"total":160658620416,"available":160641622016,"internally_reserved":0,"allocated":16998400,"data_stored":16998400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-164,"num_objects":-12,"num_object_clones":0,"num_object_copies":-24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":-338505,"num_read_kb":-11144,"num_write":-7833,"num_write_kb":-41271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002185"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1767,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-09-11T14:56:31.464779+0000","last_change":"2024-09-11T14:36:21.090039+0000","last_active":"2024-09-11T14:56:31.464779+0000","last_peered":"2024-09-11T14:56:31.464779+0000","last_clean":"2024-09-11T14:56:31.464779+0000","last_became_active":"2024-09-11T14:36:19.161959+0000","last_became_peered":"2024-09-11T14:36:19.161959+0000","last_unstale":"2024-09-11T14:56:31.464779+0000","last_undegraded":"2024-09-11T14:56:31.464779+0000","last_fullsized":"2024-09-11T14:56:31.464779+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1767,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-09-11T14:56:31.470280+0000","last_change":"2024-09-11T14:36:21.093532+0000","last_active":"2024-09-11T14:56:31.470280+0000","last_peered":"2024-09-11T14:56:31.470280+0000","last_clean":"2024-09-11T14:56:31.470280+0000","last_became_active":"2024-09-11T14:36:19.246709+0000","last_became_peered":"2024-09-11T14:36:19.246709+0000","last_unstale":"2024-09-11T14:56:31.470280+0000","last_undegraded":"2024-09-11T14:56:31.470280+0000","last_fullsized":"2024-09-11T14:56:31.470280+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1767,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-09-11T14:56:31.474047+0000","last_change":"2024-09-11T14:36:21.089054+0000","last_active":"2024-09-11T14:56:31.474047+0000","last_peered":"2024-09-11T14:56:31.474047+0000","last_clean":"2024-09-11T14:56:31.474047+0000","last_became_active":"2024-09-11T14:36:19.144086+0000","last_became_peered":"2024-09-11T14:36:19.144086+0000","last_unstale":"2024-09-11T14:56:31.474047+0000","last_undegraded":"2024-09-11T14:56:31.474047+0000","last_fullsized":"2024-09-11T14:56:31.474047+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1767,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-09-11T14:56:31.470261+0000","last_change":"2024-09-11T14:36:21.089791+0000","last_active":"2024-09-11T14:56:31.470261+0000","last_peered":"2024-09-11T14:56:31.470261+0000","last_clean":"2024-09-11T14:56:31.470261+0000","last_became_active":"2024-09-11T14:36:19.177779+0000","last_became_peered":"2024-09-11T14:36:19.177779+0000","last_unstale":"2024-09-11T14:56:31.470261+0000","last_undegraded":"2024-09-11T14:56:31.470261+0000","last_fullsized":"2024-09-11T14:56:31.470261+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":1772,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-11T14:56:44.577410+0000","last_change":"2024-09-11T14:36:21.096241+0000","last_active":"2024-09-11T14:56:44.577410+0000","last_peered":"2024-09-11T14:56:44.577410+0000","last_clean":"2024-09-11T14:56:44.577410+0000","last_became_active":"2024-09-11T14:36:19.212471+0000","last_became_peered":"2024-09-11T14:36:19.212471+0000","last_unstale":"2024-09-11T14:56:44.577410+0000","last_undegraded":"2024-09-11T14:56:44.577410+0000","last_fullsized":"2024-09-11T14:56:44.577410+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1767,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-09-11T14:56:31.519122+0000","last_change":"2024-09-11T14:36:21.094831+0000","last_active":"2024-09-11T14:56:31.519122+0000","last_peered":"2024-09-11T14:56:31.519122+0000","last_clean":"2024-09-11T14:56:31.519122+0000","last_became_active":"2024-09-11T14:36:19.234683+0000","last_became_peered":"2024-09-11T14:36:19.234683+0000","last_unstale":"2024-09-11T14:56:31.519122+0000","last_undegraded":"2024-09-11T14:56:31.519122+0000","last_fullsized":"2024-09-11T14:56:31.519122+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1767,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-09-11T14:56:31.508868+0000","last_change":"2024-09-11T14:36:21.088624+0000","last_active":"2024-09-11T14:56:31.508868+0000","last_peered":"2024-09-11T14:56:31.508868+0000","last_clean":"2024-09-11T14:56:31.508868+0000","last_became_active":"2024-09-11T14:36:19.162581+0000","last_became_peered":"2024-09-11T14:36:19.162581+0000","last_unstale":"2024-09-11T14:56:31.508868+0000","last_undegraded":"2024-09-11T14:56:31.508868+0000","last_fullsized":"2024-09-11T14:56:31.508868+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":1768,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-09-11T14:56:31.481881+0000","last_change":"2024-09-11T14:36:21.094074+0000","last_active":"2024-09-11T14:56:31.481881+0000","last_peered":"2024-09-11T14:56:31.481881+0000","last_clean":"2024-09-11T14:56:31.481881+0000","last_became_active":"2024-09-11T14:36:19.167639+0000","last_became_peered":"2024-09-11T14:36:19.167639+0000","last_unstale":"2024-09-11T14:56:31.481881+0000","last_undegraded":"2024-09-11T14:56:31.481881+0000","last_fullsized":"2024-09-11T14:56:31.481881+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":1836,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-09-11T14:56:31.461315+0000","last_change":"2024-09-11T14:36:18.038990+0000","last_active":"2024-09-11T14:56:31.461315+0000","last_peered":"2024-09-11T14:56:31.461315+0000","last_clean":"2024-09-11T14:56:31.461315+0000","last_became_active":"2024-09-11T14:36:17.997929+0000","last_became_peered":"2024-09-11T14:36:17.997929+0000","last_unstale":"2024-09-11T14:56:31.461315+0000","last_undegraded":"2024-09-11T14:56:31.461315+0000","last_fullsized":"2024-09-11T14:56:31.461315+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:16.915148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:16.915148+0000","last_clean_scrub_stamp":"2024-09-11T14:36:16.915148+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160642633728,"internally_reserved":0,"allocated":15986688,"data_stored":15986688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53545910272,"internally_reserved":0,"allocated":6963200,"data_stored":6963200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4472,"kb_used_data":4472,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293256,"statfs":{"total":53552873472,"available":53548294144,"internally_reserved":0,"allocated":4579328,"data_stored":4579328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739592,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":6860,"kb_used_data":6860,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52290868,"statfs":{"total":53552873472,"available":53545848832,"internally_reserved":0,"allocated":7024640,"data_stored":7024640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739597,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":5268,"kb_used_data":5268,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52292460,"statfs":{"total":53552873472,"available":53547479040,"internally_reserved":0,"allocated":5394432,"data_stored":5394432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53545910272,"internally_reserved":0,"allocated":6963200,"data_stored":6963200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53548228608,"internally_reserved":0,"allocated":4644864,"data_stored":4644864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53545910272,"internally_reserved":0,"allocated":6963200,"data_stored":6963200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53548494848,"internally_reserved":0,"allocated":4378624,"data_stored":4378624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:56:46.138 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T14:56:46.245 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:56:46.239+0000 7f017c6bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:56:46.248 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:56:46.246+0000 7f017c6bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:56:46.434 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:56:46.434 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":599,"fsid":"1c448697-f86d-4ae7-8a96-5392aad74a40","created":"2024-09-11T14:36:08.010044+0000","modified":"2024-09-11T14:56:44.353407+0000","last_up_change":"2024-09-11T14:36:15.903577+0000","last_in_change":"2024-09-11T14:36:09.898291+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T14:36:16.855699+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T14:36:17.783600+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d19dbbd7-da6f-45c6-8e59-2023cb4b8262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3554493692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6807","nonce":3554493692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":3554493692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6811","nonce":3554493692}]},"public_addr":"172.21.15.188:6806/3554493692","cluster_addr":"172.21.15.188:6807/3554493692","heartbeat_back_addr":"172.21.15.188:6812/3554493692","heartbeat_front_addr":"172.21.15.188:6811/3554493692","state":["exists","up"]},{"osd":1,"uuid":"e977da2d-0945-4956-8491-322cd072ca16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6803","nonce":1156335168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":1156335168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":1156335168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6805","nonce":1156335168}]},"public_addr":"172.21.15.188:6803/1156335168","cluster_addr":"172.21.15.188:6804/1156335168","heartbeat_back_addr":"172.21.15.188:6802/1156335168","heartbeat_front_addr":"172.21.15.188:6805/1156335168","state":["exists","up"]},{"osd":2,"uuid":"8bb79e5a-9a64-4df7-9335-0cf03ab348e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":847045712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6809","nonce":847045712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":847045712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":847045712}]},"public_addr":"172.21.15.188:6810/847045712","cluster_addr":"172.21.15.188:6809/847045712","heartbeat_back_addr":"172.21.15.188:6813/847045712","heartbeat_front_addr":"172.21.15.188:6808/847045712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]},{"pool":3,"snaps":[{"begin":2,"length":298}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:56:47.449 INFO:tasks.ceph:Scrubbing osd.0 2024-09-11T14:56:47.449 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-11T14:56:47.576 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:56:47.565+0000 7fe3fe391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:56:47.579 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:56:47.577+0000 7fe3fe391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:56:47.594 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-11T14:56:47.595 INFO:teuthology.orchestra.run.smithi188.stdout: "success": "" 2024-09-11T14:56:47.595 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-11T14:56:47.606 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-11T14:56:47.716 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:56:47.709+0000 7f097c591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:56:47.719 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:56:47.717+0000 7f097c591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:56:47.905 INFO:teuthology.orchestra.run.smithi188.stderr:instructed osd(s) 0 to deep-scrub 2024-09-11T14:56:47.922 INFO:tasks.ceph:Scrubbing osd.1 2024-09-11T14:56:47.922 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-11T14:56:48.034 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:56:48.023+0000 7fa2eff91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:56:48.038 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:56:48.035+0000 7fa2eff91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:56:48.052 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-11T14:56:48.052 INFO:teuthology.orchestra.run.smithi188.stdout: "success": "" 2024-09-11T14:56:48.053 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-11T14:56:48.064 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-11T14:56:48.173 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:56:48.165+0000 7fb463391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:56:48.176 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:56:48.174+0000 7fb463391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:56:48.361 INFO:teuthology.orchestra.run.smithi188.stderr:instructed osd(s) 1 to deep-scrub 2024-09-11T14:56:48.382 INFO:tasks.ceph:Scrubbing osd.2 2024-09-11T14:56:48.382 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-11T14:56:48.501 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:56:48.491+0000 7f67dc263640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:56:48.505 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:56:48.503+0000 7f67dc263640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:56:48.520 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-11T14:56:48.520 INFO:teuthology.orchestra.run.smithi188.stdout: "success": "" 2024-09-11T14:56:48.520 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-11T14:56:48.532 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-11T14:56:48.641 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:56:48.634+0000 7f6567c88640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:56:48.644 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:56:48.642+0000 7f6567c88640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:56:48.831 INFO:teuthology.orchestra.run.smithi188.stderr:instructed osd(s) 2 to deep-scrub 2024-09-11T14:56:48.851 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:56:48.958 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:56:48.953+0000 7f7bc4f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:56:48.962 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:56:48.960+0000 7f7bc4f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:56:49.142 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:56:49.143 INFO:teuthology.orchestra.run.smithi188.stderr:dumped all 2024-09-11T14:56:49.155 INFO:teuthology.orchestra.run.smithi188.stdout:{"pg_ready":true,"pg_map":{"version":1210,"stamp":"2024-09-11T14:56:49.083345+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":10352,"kb_used_data":10352,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156882832,"statfs":{"total":160658620416,"available":160648019968,"internally_reserved":0,"allocated":10600448,"data_stored":10600448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-112,"num_objects":-11,"num_object_clones":0,"num_object_copies":-22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":-338977,"num_read_kb":-11211,"num_write":-7885,"num_write_kb":-41290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002053"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1770,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-11T14:56:44.532763+0000","last_change":"2024-09-11T14:36:21.090039+0000","last_active":"2024-09-11T14:56:44.532763+0000","last_peered":"2024-09-11T14:56:44.532763+0000","last_clean":"2024-09-11T14:56:44.532763+0000","last_became_active":"2024-09-11T14:36:19.161959+0000","last_became_peered":"2024-09-11T14:36:19.161959+0000","last_unstale":"2024-09-11T14:56:44.532763+0000","last_undegraded":"2024-09-11T14:56:44.532763+0000","last_fullsized":"2024-09-11T14:56:44.532763+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1770,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-11T14:56:44.558999+0000","last_change":"2024-09-11T14:36:21.093532+0000","last_active":"2024-09-11T14:56:44.558999+0000","last_peered":"2024-09-11T14:56:44.558999+0000","last_clean":"2024-09-11T14:56:44.558999+0000","last_became_active":"2024-09-11T14:36:19.246709+0000","last_became_peered":"2024-09-11T14:36:19.246709+0000","last_unstale":"2024-09-11T14:56:44.558999+0000","last_undegraded":"2024-09-11T14:56:44.558999+0000","last_fullsized":"2024-09-11T14:56:44.558999+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1770,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-11T14:56:44.524835+0000","last_change":"2024-09-11T14:36:21.089054+0000","last_active":"2024-09-11T14:56:44.524835+0000","last_peered":"2024-09-11T14:56:44.524835+0000","last_clean":"2024-09-11T14:56:44.524835+0000","last_became_active":"2024-09-11T14:36:19.144086+0000","last_became_peered":"2024-09-11T14:36:19.144086+0000","last_unstale":"2024-09-11T14:56:44.524835+0000","last_undegraded":"2024-09-11T14:56:44.524835+0000","last_fullsized":"2024-09-11T14:56:44.524835+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1770,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-11T14:56:44.513898+0000","last_change":"2024-09-11T14:36:21.089791+0000","last_active":"2024-09-11T14:56:44.513898+0000","last_peered":"2024-09-11T14:56:44.513898+0000","last_clean":"2024-09-11T14:56:44.513898+0000","last_became_active":"2024-09-11T14:36:19.177779+0000","last_became_peered":"2024-09-11T14:36:19.177779+0000","last_unstale":"2024-09-11T14:56:44.513898+0000","last_undegraded":"2024-09-11T14:56:44.513898+0000","last_fullsized":"2024-09-11T14:56:44.513898+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":1772,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-11T14:56:44.577410+0000","last_change":"2024-09-11T14:36:21.096241+0000","last_active":"2024-09-11T14:56:44.577410+0000","last_peered":"2024-09-11T14:56:44.577410+0000","last_clean":"2024-09-11T14:56:44.577410+0000","last_became_active":"2024-09-11T14:36:19.212471+0000","last_became_peered":"2024-09-11T14:36:19.212471+0000","last_unstale":"2024-09-11T14:56:44.577410+0000","last_undegraded":"2024-09-11T14:56:44.577410+0000","last_fullsized":"2024-09-11T14:56:44.577410+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1770,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-11T14:56:44.625517+0000","last_change":"2024-09-11T14:36:21.094831+0000","last_active":"2024-09-11T14:56:44.625517+0000","last_peered":"2024-09-11T14:56:44.625517+0000","last_clean":"2024-09-11T14:56:44.625517+0000","last_became_active":"2024-09-11T14:36:19.234683+0000","last_became_peered":"2024-09-11T14:36:19.234683+0000","last_unstale":"2024-09-11T14:56:44.625517+0000","last_undegraded":"2024-09-11T14:56:44.625517+0000","last_fullsized":"2024-09-11T14:56:44.625517+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1770,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-11T14:56:44.574854+0000","last_change":"2024-09-11T14:36:21.088624+0000","last_active":"2024-09-11T14:56:44.574854+0000","last_peered":"2024-09-11T14:56:44.574854+0000","last_clean":"2024-09-11T14:56:44.574854+0000","last_became_active":"2024-09-11T14:36:19.162581+0000","last_became_peered":"2024-09-11T14:36:19.162581+0000","last_unstale":"2024-09-11T14:56:44.574854+0000","last_undegraded":"2024-09-11T14:56:44.574854+0000","last_fullsized":"2024-09-11T14:56:44.574854+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":1771,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-11T14:56:44.549009+0000","last_change":"2024-09-11T14:36:21.094074+0000","last_active":"2024-09-11T14:56:44.549009+0000","last_peered":"2024-09-11T14:56:44.549009+0000","last_clean":"2024-09-11T14:56:44.549009+0000","last_became_active":"2024-09-11T14:36:19.167639+0000","last_became_peered":"2024-09-11T14:36:19.167639+0000","last_unstale":"2024-09-11T14:56:44.549009+0000","last_undegraded":"2024-09-11T14:56:44.549009+0000","last_fullsized":"2024-09-11T14:56:44.549009+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:17.919163+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":1839,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-11T14:56:44.530262+0000","last_change":"2024-09-11T14:36:18.038990+0000","last_active":"2024-09-11T14:56:44.530262+0000","last_peered":"2024-09-11T14:56:44.530262+0000","last_clean":"2024-09-11T14:56:44.530262+0000","last_became_active":"2024-09-11T14:36:17.997929+0000","last_became_peered":"2024-09-11T14:36:17.997929+0000","last_unstale":"2024-09-11T14:56:44.530262+0000","last_undegraded":"2024-09-11T14:56:44.530262+0000","last_fullsized":"2024-09-11T14:56:44.530262+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:36:16.915148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:36:16.915148+0000","last_clean_scrub_stamp":"2024-09-11T14:36:16.915148+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160650403840,"internally_reserved":0,"allocated":8216576,"data_stored":8216576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53550485504,"internally_reserved":0,"allocated":2387968,"data_stored":2387968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1156,"kb_used_data":1156,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296572,"statfs":{"total":53552873472,"available":53551689728,"internally_reserved":0,"allocated":1183744,"data_stored":1183744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739597,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3928,"kb_used_data":3928,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293800,"statfs":{"total":53552873472,"available":53548851200,"internally_reserved":0,"allocated":4022272,"data_stored":4022272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739597,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":5268,"kb_used_data":5268,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52292460,"statfs":{"total":53552873472,"available":53547479040,"internally_reserved":0,"allocated":5394432,"data_stored":5394432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53550485504,"internally_reserved":0,"allocated":2387968,"data_stored":2387968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53548228608,"internally_reserved":0,"allocated":4644864,"data_stored":4644864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550485504,"internally_reserved":0,"allocated":2387968,"data_stored":2387968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53551689728,"internally_reserved":0,"allocated":1183744,"data_stored":1183744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:56:49.159 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-11T14:36:17.919163+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=36, tm_sec=17, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=56, tm_sec=46, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:56:49.159 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-11T14:36:17.919163+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=36, tm_sec=17, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=56, tm_sec=46, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:56:49.159 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-11T14:36:17.919163+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=36, tm_sec=17, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=56, tm_sec=46, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:56:49.159 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-11T14:36:17.919163+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=36, tm_sec=17, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=56, tm_sec=46, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:56:49.159 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-11T14:36:17.919163+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=36, tm_sec=17, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=56, tm_sec=46, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:56:49.159 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-11T14:36:17.919163+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=36, tm_sec=17, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=56, tm_sec=46, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:56:49.160 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-11T14:36:17.919163+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=36, tm_sec=17, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=56, tm_sec=46, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:56:49.160 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-11T14:36:17.919163+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=36, tm_sec=17, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=56, tm_sec=46, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:56:49.160 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-11T14:36:16.915148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=36, tm_sec=16, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=56, tm_sec=46, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:56:49.160 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-11T14:57:09.161 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:57:09.284 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:57:09.279+0000 7fb65bca3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:57:09.288 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:57:09.287+0000 7fb65bca3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:57:09.472 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-11T14:57:09.473 INFO:teuthology.orchestra.run.smithi188.stderr:dumped all 2024-09-11T14:57:09.485 INFO:teuthology.orchestra.run.smithi188.stdout:{"pg_ready":true,"pg_map":{"version":1220,"stamp":"2024-09-11T14:57:09.088030+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":5544,"kb_used_data":5544,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156887640,"statfs":{"total":160658620416,"available":160652943360,"internally_reserved":0,"allocated":5677056,"data_stored":5677056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003024"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1772,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-11T14:56:48.373001+0000","last_change":"2024-09-11T14:56:48.420507+0000","last_active":"2024-09-11T14:56:48.373001+0000","last_peered":"2024-09-11T14:56:48.373001+0000","last_clean":"2024-09-11T14:56:48.373001+0000","last_became_active":"2024-09-11T14:36:19.161959+0000","last_became_peered":"2024-09-11T14:36:19.161959+0000","last_unstale":"2024-09-11T14:56:48.373001+0000","last_undegraded":"2024-09-11T14:56:48.373001+0000","last_fullsized":"2024-09-11T14:56:48.373001+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:56:48.420336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:56:48.420336+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1772,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-11T14:56:48.374334+0000","last_change":"2024-09-11T14:56:48.401469+0000","last_active":"2024-09-11T14:56:48.374334+0000","last_peered":"2024-09-11T14:56:48.374334+0000","last_clean":"2024-09-11T14:56:48.374334+0000","last_became_active":"2024-09-11T14:36:19.246709+0000","last_became_peered":"2024-09-11T14:36:19.246709+0000","last_unstale":"2024-09-11T14:56:48.374334+0000","last_undegraded":"2024-09-11T14:56:48.374334+0000","last_fullsized":"2024-09-11T14:56:48.374334+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:56:48.401293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:56:48.401293+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1772,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-11T14:56:48.370024+0000","last_change":"2024-09-11T14:56:48.414202+0000","last_active":"2024-09-11T14:56:48.370024+0000","last_peered":"2024-09-11T14:56:48.370024+0000","last_clean":"2024-09-11T14:56:48.370024+0000","last_became_active":"2024-09-11T14:36:19.144086+0000","last_became_peered":"2024-09-11T14:36:19.144086+0000","last_unstale":"2024-09-11T14:56:48.370024+0000","last_undegraded":"2024-09-11T14:56:48.370024+0000","last_fullsized":"2024-09-11T14:56:48.370024+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:56:48.414022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:56:48.414022+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1772,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-11T14:56:48.366797+0000","last_change":"2024-09-11T14:56:48.417681+0000","last_active":"2024-09-11T14:56:48.366797+0000","last_peered":"2024-09-11T14:56:48.366797+0000","last_clean":"2024-09-11T14:56:48.366797+0000","last_became_active":"2024-09-11T14:36:19.177779+0000","last_became_peered":"2024-09-11T14:36:19.177779+0000","last_unstale":"2024-09-11T14:56:48.366797+0000","last_undegraded":"2024-09-11T14:56:48.366797+0000","last_fullsized":"2024-09-11T14:56:48.366797+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:56:48.417506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:56:48.417506+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":1774,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-11T14:56:47.906429+0000","last_change":"2024-09-11T14:56:47.924604+0000","last_active":"2024-09-11T14:56:47.906429+0000","last_peered":"2024-09-11T14:56:47.906429+0000","last_clean":"2024-09-11T14:56:47.906429+0000","last_became_active":"2024-09-11T14:36:19.212471+0000","last_became_peered":"2024-09-11T14:36:19.212471+0000","last_unstale":"2024-09-11T14:56:47.906429+0000","last_undegraded":"2024-09-11T14:56:47.906429+0000","last_fullsized":"2024-09-11T14:56:47.906429+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"14'2","last_scrub_stamp":"2024-09-11T14:56:47.924413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:56:47.924413+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1772,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-11T14:56:48.833873+0000","last_change":"2024-09-11T14:56:48.848876+0000","last_active":"2024-09-11T14:56:48.833873+0000","last_peered":"2024-09-11T14:56:48.833873+0000","last_clean":"2024-09-11T14:56:48.833873+0000","last_became_active":"2024-09-11T14:36:19.234683+0000","last_became_peered":"2024-09-11T14:36:19.234683+0000","last_unstale":"2024-09-11T14:56:48.833873+0000","last_undegraded":"2024-09-11T14:56:48.833873+0000","last_fullsized":"2024-09-11T14:56:48.833873+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:56:48.848643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:56:48.848643+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1772,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-11T14:56:48.832371+0000","last_change":"2024-09-11T14:56:48.845941+0000","last_active":"2024-09-11T14:56:48.832371+0000","last_peered":"2024-09-11T14:56:48.832371+0000","last_clean":"2024-09-11T14:56:48.832371+0000","last_became_active":"2024-09-11T14:36:19.162581+0000","last_became_peered":"2024-09-11T14:36:19.162581+0000","last_unstale":"2024-09-11T14:56:48.832371+0000","last_undegraded":"2024-09-11T14:56:48.832371+0000","last_fullsized":"2024-09-11T14:56:48.832371+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:56:48.845751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:56:48.845751+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":1773,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-11T14:56:48.366759+0000","last_change":"2024-09-11T14:56:48.402775+0000","last_active":"2024-09-11T14:56:48.366759+0000","last_peered":"2024-09-11T14:56:48.366759+0000","last_clean":"2024-09-11T14:56:48.366759+0000","last_became_active":"2024-09-11T14:36:19.167639+0000","last_became_peered":"2024-09-11T14:36:19.167639+0000","last_unstale":"2024-09-11T14:56:48.366759+0000","last_undegraded":"2024-09-11T14:56:48.366759+0000","last_fullsized":"2024-09-11T14:56:48.366759+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"12'1","last_scrub_stamp":"2024-09-11T14:56:48.402598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:56:48.402598+0000","last_clean_scrub_stamp":"2024-09-11T14:36:17.919163+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":1841,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-11T14:56:48.363815+0000","last_change":"2024-09-11T14:56:48.438312+0000","last_active":"2024-09-11T14:56:48.363815+0000","last_peered":"2024-09-11T14:56:48.363815+0000","last_clean":"2024-09-11T14:56:48.363815+0000","last_became_active":"2024-09-11T14:36:17.997929+0000","last_became_peered":"2024-09-11T14:36:17.997929+0000","last_unstale":"2024-09-11T14:56:48.363815+0000","last_undegraded":"2024-09-11T14:56:48.363815+0000","last_fullsized":"2024-09-11T14:56:48.363815+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"12'52","last_scrub_stamp":"2024-09-11T14:56:48.438098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:56:48.438098+0000","last_clean_scrub_stamp":"2024-09-11T14:36:16.915148+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160652943360,"internally_reserved":0,"allocated":5677056,"data_stored":5677056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53550485504,"internally_reserved":0,"allocated":2387968,"data_stored":2387968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1156,"kb_used_data":1156,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296572,"statfs":{"total":53552873472,"available":53551689728,"internally_reserved":0,"allocated":1183744,"data_stored":1183744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739617,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2332,"kb_used_data":2332,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295396,"statfs":{"total":53552873472,"available":53550485504,"internally_reserved":0,"allocated":2387968,"data_stored":2387968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739617,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2056,"kb_used_data":2056,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295672,"statfs":{"total":53552873472,"available":53550768128,"internally_reserved":0,"allocated":2105344,"data_stored":2105344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53550485504,"internally_reserved":0,"allocated":2387968,"data_stored":2387968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53550768128,"internally_reserved":0,"allocated":2105344,"data_stored":2105344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550485504,"internally_reserved":0,"allocated":2387968,"data_stored":2387968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53551689728,"internally_reserved":0,"allocated":1183744,"data_stored":1183744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:57:09.486 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-11T14:57:09.593 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:57:09.585+0000 7fbf53948640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:57:09.597 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-11T14:57:09.596+0000 7fbf53948640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:57:09.805 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-11T14:57:09.805 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-11T14:57:09.806 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-11T14:57:09.806 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T14:57:09.881 INFO:tasks.ceph.osd.0:Stopped 2024-09-11T14:57:09.882 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-11T14:57:09.882 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T14:57:09.950 INFO:tasks.ceph.osd.1:Stopped 2024-09-11T14:57:09.950 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-11T14:57:09.951 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T14:57:10.009 INFO:tasks.ceph.osd.2:Stopped 2024-09-11T14:57:10.010 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-11T14:57:10.010 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-11T14:57:10.010 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T14:57:10.039 INFO:tasks.ceph.mgr.x:Stopped 2024-09-11T14:57:10.039 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-11T14:57:10.040 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T14:57:10.040 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T14:57:10.094 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T14:57:10.095 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-11T14:57:10.095 DEBUG:teuthology.orchestra.run.smithi188:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-09-11T14:57:10.161 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi188.front.sepia.ceph.com 2024-09-11T14:57:10.161 DEBUG:teuthology.orchestra.run.smithi188:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-11T14:57:10.450 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi188.front.sepia.ceph.com 2024-09-11T14:57:10.450 DEBUG:teuthology.orchestra.run.smithi188:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-11T14:57:10.533 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi188.front.sepia.ceph.com 2024-09-11T14:57:10.533 DEBUG:teuthology.orchestra.run.smithi188:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-11T14:57:10.580 INFO:tasks.ceph:Archiving mon data... 2024-09-11T14:57:10.581 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901368/data/mon.a.tgz 2024-09-11T14:57:10.582 DEBUG:teuthology.orchestra.run.smithi188:> mktemp 2024-09-11T14:57:10.596 INFO:teuthology.orchestra.run.smithi188.stdout:/tmp/tmp.GdtKhf46jD 2024-09-11T14:57:10.596 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.GdtKhf46jD 2024-09-11T14:57:10.713 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 0666 /tmp/tmp.GdtKhf46jD 2024-09-11T14:57:10.795 DEBUG:teuthology.orchestra.remote:smithi188:/tmp/tmp.GdtKhf46jD is 242KB 2024-09-11T14:57:10.889 DEBUG:teuthology.orchestra.run.smithi188:> rm -fr /tmp/tmp.GdtKhf46jD 2024-09-11T14:57:10.904 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-11T14:57:10.904 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T14:57:11.023 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-11T14:57:11.023 INFO:tasks.ceph:Archiving crash dumps... 2024-09-11T14:57:11.025 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901368/remote/smithi188/crash 2024-09-11T14:57:11.026 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-11T14:57:11.087 INFO:tasks.ceph:Compressing logs... 2024-09-11T14:57:11.087 DEBUG:teuthology.orchestra.run.smithi188:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T14:57:11.153 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-11T14:57:11.153 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28580.log 2024-09-11T14:57:11.153 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26072.log 2024-09-11T14:57:11.153 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.28580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27211.log 2024-09-11T14:57:11.153 INFO:teuthology.orchestra.run.smithi188.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28580.log.gz 2024-09-11T14:57:11.154 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.26072.log: gzip 50.0% -- replaced with /var/log/ceph/ceph-client.admin.26072.log.gz -5 2024-09-11T14:57:11.154 INFO:teuthology.orchestra.run.smithi188.stderr: --verbose -- /var/log/ceph/ceph-client.admin.27172.log 2024-09-11T14:57:11.154 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.27211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26794.log 2024-09-11T14:57:11.154 INFO:teuthology.orchestra.run.smithi188.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.27211.log.gz 2024-09-11T14:57:11.154 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.27172.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-11T14:57:11.154 INFO:teuthology.orchestra.run.smithi188.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.27172.log.gz 2024-09-11T14:57:11.154 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.26794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29042.log 2024-09-11T14:57:11.154 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26601.log 2024-09-11T14:57:11.154 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26836.log 2024-09-11T14:57:11.169 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28748.log 2024-09-11T14:57:11.169 INFO:teuthology.orchestra.run.smithi188.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.26794.log.gz 2024-09-11T14:57:11.170 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.29042.log: /var/log/ceph/ceph-client.admin.26601.log: /var/log/ceph/ceph-client.admin.26836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28874.log 2024-09-11T14:57:11.170 INFO:teuthology.orchestra.run.smithi188.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29042.log.gz 2024-09-11T14:57:11.170 INFO:teuthology.orchestra.run.smithi188.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.26601.log.gz 2024-09-11T14:57:11.170 INFO:teuthology.orchestra.run.smithi188.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.26836.log.gz 2024-09-11T14:57:11.170 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.28748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28958.log 2024-09-11T14:57:11.170 INFO:teuthology.orchestra.run.smithi188.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28748.log.gz 2024-09-11T14:57:11.170 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.28874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26182.log 2024-09-11T14:57:11.170 INFO:teuthology.orchestra.run.smithi188.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28874.log.gz 2024-09-11T14:57:11.170 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.28958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27382.log 2024-09-11T14:57:11.170 INFO:teuthology.orchestra.run.smithi188.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28958.log.gz 2024-09-11T14:57:11.170 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.26182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26657.log 2024-09-11T14:57:11.170 INFO:teuthology.orchestra.run.smithi188.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.26182.log.gz 2024-09-11T14:57:11.170 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.27382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28916.log 2024-09-11T14:57:11.171 INFO:teuthology.orchestra.run.smithi188.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.27382.log.gz 2024-09-11T14:57:11.171 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.26657.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.26657.log.gz 2024-09-11T14:57:11.171 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-11T14:57:11.171 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.28916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28832.log 2024-09-11T14:57:11.171 INFO:teuthology.orchestra.run.smithi188.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.28916.log.gz 2024-09-11T14:57:11.171 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26682.log 2024-09-11T14:57:11.171 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.28832.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28832.log.gz 2024-09-11T14:57:11.171 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26222.log 2024-09-11T14:57:11.171 INFO:teuthology.orchestra.run.smithi188.stderr: 86.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-11T14:57:11.171 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.26682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27601.log 2024-09-11T14:57:11.171 INFO:teuthology.orchestra.run.smithi188.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.26682.log.gz 2024-09-11T14:57:11.171 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.26222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27340.log 2024-09-11T14:57:11.171 INFO:teuthology.orchestra.run.smithi188.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.26222.log.gz 2024-09-11T14:57:11.171 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.27601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27298.log 2024-09-11T14:57:11.171 INFO:teuthology.orchestra.run.smithi188.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.27601.log.gz 2024-09-11T14:57:11.172 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.27340.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.27340.log.gz 2024-09-11T14:57:11.172 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26273.log 2024-09-11T14:57:11.172 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26752.log 2024-09-11T14:57:11.172 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.27298.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27298.log.gz 2024-09-11T14:57:11.172 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.26273.log: gzip -5 --verbose -- 50.0% /var/log/ceph/ceph.tmp-client.admin.25034.log -- replaced with /var/log/ceph/ceph-client.admin.26273.log.gz 2024-09-11T14:57:11.172 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-09-11T14:57:11.172 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.26752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27086.log 2024-09-11T14:57:11.172 INFO:teuthology.orchestra.run.smithi188.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.26752.log.gz 2024-09-11T14:57:11.172 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph.tmp-client.admin.25034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28622.log 2024-09-11T14:57:11.172 INFO:teuthology.orchestra.run.smithi188.stderr: 66.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.25034.log.gz 2024-09-11T14:57:11.172 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.27086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25984.log 2024-09-11T14:57:11.172 INFO:teuthology.orchestra.run.smithi188.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.27086.log.gz 2024-09-11T14:57:11.172 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.28622.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-11T14:57:11.172 INFO:teuthology.orchestra.run.smithi188.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28622.log.gz 2024-09-11T14:57:11.173 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.25984.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-11T14:57:11.173 INFO:teuthology.orchestra.run.smithi188.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.25984.log.gz 2024-09-11T14:57:11.173 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26461.log 2024-09-11T14:57:11.173 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26960.log 2024-09-11T14:57:11.173 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.26461.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-11T14:57:11.173 INFO:teuthology.orchestra.run.smithi188.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.26461.log.gz 2024-09-11T14:57:11.173 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.26960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28664.log 2024-09-11T14:57:11.173 INFO:teuthology.orchestra.run.smithi188.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.26960.log.gz 2024-09-11T14:57:11.173 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29000.log 2024-09-11T14:57:11.173 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.28664.log: gzip -5 --verbose 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28664.log.gz 2024-09-11T14:57:11.173 INFO:teuthology.orchestra.run.smithi188.stderr: -- /var/log/ceph/ceph-mgr.x.log 2024-09-11T14:57:11.173 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.29000.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29000.log.gz 2024-09-11T14:57:11.173 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27212.log 2024-09-11T14:57:11.174 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26958.log 2024-09-11T14:57:11.174 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.27212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28706.log 2024-09-11T14:57:11.174 INFO:teuthology.orchestra.run.smithi188.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.27212.log.gz 2024-09-11T14:57:11.174 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.26958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26878.log 2024-09-11T14:57:11.174 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.28706.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.26958.log.gz 2024-09-11T14:57:11.174 INFO:teuthology.orchestra.run.smithi188.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28706.log.gz 2024-09-11T14:57:11.174 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27046.log 2024-09-11T14:57:11.174 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.26878.log: 86.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-11T14:57:11.174 INFO:teuthology.orchestra.run.smithi188.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.26878.log.gz 2024-09-11T14:57:11.174 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26431.log 2024-09-11T14:57:11.174 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.27046.log: /var/log/ceph/ceph-client.admin.28790.log 2024-09-11T14:57:11.174 INFO:teuthology.orchestra.run.smithi188.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27046.log.gz/var/log/ceph/ceph-client.admin.26431.log: 2024-09-11T14:57:11.174 INFO:teuthology.orchestra.run.smithi188.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.26431.log.gz 2024-09-11T14:57:11.174 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26959.log 2024-09-11T14:57:11.175 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.28790.log: gzip -5 --verbose -- 49.6% /var/log/ceph/ceph-client.admin.27085.log 2024-09-11T14:57:11.175 INFO:teuthology.orchestra.run.smithi188.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28790.log.gz 2024-09-11T14:57:11.175 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.26959.log: 50.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29082.log 2024-09-11T14:57:11.175 INFO:teuthology.orchestra.run.smithi188.stderr: -- replaced with /var/log/ceph/ceph-client.admin.26959.log.gz 2024-09-11T14:57:11.175 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.27085.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.27085.log.gz 2024-09-11T14:57:11.175 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.29082.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29082.log.gz 2024-09-11T14:57:11.306 INFO:teuthology.orchestra.run.smithi188.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-11T14:57:11.674 INFO:teuthology.orchestra.run.smithi188.stderr: 89.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-11T14:57:14.985 INFO:teuthology.orchestra.run.smithi188.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-11T14:57:16.556 INFO:teuthology.orchestra.run.smithi188.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-11T14:57:19.804 INFO:teuthology.orchestra.run.smithi188.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-11T14:57:19.806 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-09-11T14:57:19.806 INFO:teuthology.orchestra.run.smithi188.stderr:real 0m8.664s 2024-09-11T14:57:19.806 INFO:teuthology.orchestra.run.smithi188.stderr:user 0m17.814s 2024-09-11T14:57:19.806 INFO:teuthology.orchestra.run.smithi188.stderr:sys 0m0.741s 2024-09-11T14:57:19.807 INFO:tasks.ceph:Archiving logs... 2024-09-11T14:57:19.807 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/log/ceph to /home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901368/remote/smithi188/log 2024-09-11T14:57:19.808 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T14:57:22.669 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-11T14:57:22.678 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-11T14:57:22.678 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T14:57:22.744 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-11T14:57:22.753 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-11T14:57:22.753 DEBUG:teuthology.orchestra.run.smithi188:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T14:57:22.774 INFO:teuthology.orchestra.run.smithi188.stderr:bash: line 1: ntpq: command not found 2024-09-11T14:57:22.778 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T14:57:22.779 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2024-09-11T14:57:22.779 INFO:teuthology.orchestra.run.smithi188.stdout:^* hv01.front.sepia.ceph.com 3 6 377 50 +1100us[+1089us] +/- 63ms 2024-09-11T14:57:22.779 INFO:teuthology.orchestra.run.smithi188.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 46 -613us[ -613us] +/- 62ms 2024-09-11T14:57:22.779 INFO:teuthology.orchestra.run.smithi188.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 -526us[ -536us] +/- 61ms 2024-09-11T14:57:22.779 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T14:57:22.780 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-11T14:57:22.789 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-11T14:57:22.789 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-11T14:57:22.799 DEBUG:teuthology.orchestra.run.smithi188:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-11T14:57:22.885 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T14:57:22.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T14:57:22.913 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-11T14:57:22.922 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-11T14:57:22.933 INFO:teuthology.task.internal:Duration was 1632.342690 seconds 2024-09-11T14:57:22.933 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-11T14:57:22.943 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-11T14:57:22.944 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T14:57:22.993 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T14:57:23.416 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-11T14:57:23.416 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi188.front.sepia.ceph.com 2024-09-11T14:57:23.417 DEBUG:teuthology.orchestra.run.smithi188:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T14:57:23.477 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-11T14:57:23.477 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T14:57:23.544 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-11T14:57:23.544 DEBUG:teuthology.orchestra.run.smithi188:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T14:57:23.681 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-11T14:57:23.690 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-11T14:57:23.691 DEBUG:teuthology.orchestra.run.smithi188:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T14:57:23.715 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-11T14:57:23.725 DEBUG:teuthology.orchestra.run.smithi188:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T14:57:23.780 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = core 2024-09-11T14:57:23.791 DEBUG:teuthology.orchestra.run.smithi188:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T14:57:23.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T14:57:23.846 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-11T14:57:23.860 INFO:teuthology.task.internal:Transferring archived files... 2024-09-11T14:57:23.861 DEBUG:teuthology.misc:Transferring archived files from smithi188:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901368/remote/smithi188 2024-09-11T14:57:23.861 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T14:57:23.918 INFO:teuthology.task.internal:Removing archive directory... 2024-09-11T14:57:23.918 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T14:57:23.969 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-11T14:57:23.990 INFO:teuthology.task.internal:Not uploading archives. 2024-09-11T14:57:23.990 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-11T14:57:24.001 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-11T14:57:24.002 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T14:57:24.023 INFO:teuthology.orchestra.run.smithi188.stdout: 265861 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 14:57 /home/ubuntu/cephtest 2024-09-11T14:57:24.025 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-11T14:57:24.035 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-11T14:57:24.069 INFO:teuthology.run:Summary data: description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_python_api_tests_old_format} duration: 1632.3426904678345 flavor: crimson owner: scheduled_matan@teuthology success: true 2024-09-11T14:57:24.069 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T14:57:24.165 INFO:teuthology.run:pass