2024-03-27T08:33:33.989 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-27T08:33:33.989 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/matan-2024-03-27_08:05:55-crimson-rados-wip-matanb-crimson-foreign-ptr-distro-crimson-smithi/7625904 --name matan-2024-03-27_08:05:55-crimson-rados-wip-matanb-crimson-foreign-ptr-distro-crimson-smithi --owner scheduled_matan@teuthology --verbose --description crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_lock_and_fence} -- /home/teuthworker/archive/matan-2024-03-27_08:05:55-crimson-rados-wip-matanb-crimson-foreign-ptr-distro-crimson-smithi/7625904/orig.config.yaml 2024-03-27T08:33:34.017 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-27T08:33:34.103 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2024-03-27_08:05:55-crimson-rados-wip-matanb-crimson-foreign-ptr-distro-crimson-smithi/7625904 branch: wip-matanb-crimson-foreign-ptr description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_lock_and_fence} email: null first_in_suite: false job_id: '7625904' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2024-03-27_08:05:55-crimson-rados-wip-matanb-crimson-foreign-ptr-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-matanb-crimson-foreign-ptr ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd pool default crimson: true osd: crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug ms: 1 debug osd: 20 osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-1 osd.1: crimson seastar cpu cores: 2-3 osd.2: crimson seastar cpu cores: 4-5 flavor: crimson log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: 022fc18aa97a30a2f3803bcc4f7d8f3544371a09 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: crimson sha1: 022fc18aa97a30a2f3803bcc4f7d8f3544371a09 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: eae914f097640d21427554e7f6aa6f1e9c4a7748 owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 7338 sha1: 022fc18aa97a30a2f3803bcc4f7d8f3544371a09 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_eae914f097640d21427554e7f6aa6f1e9c4a7748/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: eae914f097640d21427554e7f6aa6f1e9c4a7748 targets: smithi183.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKX/Lm8n4pB52qJdUdfLbrooLRiRZrAIg+jORRiQmF+LaN66a4DgOu9uW2B6A2tbD9+uN1ol8rmfSmTXPcjkVyA= tasks: - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 flavor: crimson - workunit: clients: client.0: - rbd/test_lock_fence.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-27_08:05:55 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-27T08:33:34.104 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_eae914f097640d21427554e7f6aa6f1e9c4a7748/qa; will attempt to use it 2024-03-27T08:33:34.104 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_eae914f097640d21427554e7f6aa6f1e9c4a7748/qa/tasks 2024-03-27T08:33:34.104 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-27T08:33:34.106 INFO:teuthology.task.internal:Checking packages... 2024-03-27T08:33:34.129 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '022fc18aa97a30a2f3803bcc4f7d8f3544371a09' 2024-03-27T08:33:34.129 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-27T08:33:34.129 INFO:teuthology.packaging:ref: None 2024-03-27T08:33:34.129 INFO:teuthology.packaging:tag: None 2024-03-27T08:33:34.129 INFO:teuthology.packaging:branch: wip-matanb-crimson-foreign-ptr 2024-03-27T08:33:34.129 INFO:teuthology.packaging:sha1: 022fc18aa97a30a2f3803bcc4f7d8f3544371a09 2024-03-27T08:33:34.130 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-matanb-crimson-foreign-ptr 2024-03-27T08:33:34.348 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2463.g022fc18a 2024-03-27T08:33:34.349 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-27T08:33:34.356 INFO:teuthology.task.internal:no buildpackages task found 2024-03-27T08:33:34.356 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-27T08:33:34.372 INFO:teuthology.task.internal:Saving configuration 2024-03-27T08:33:34.382 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-27T08:33:34.388 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-27T08:33:34.413 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-03-27_08:05:55-crimson-rados-wip-matanb-crimson-foreign-ptr-distro-crimson-smithi/7625904', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-27 08:28:41.389800', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKX/Lm8n4pB52qJdUdfLbrooLRiRZrAIg+jORRiQmF+LaN66a4DgOu9uW2B6A2tbD9+uN1ol8rmfSmTXPcjkVyA='} 2024-03-27T08:33:34.414 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-27T08:33:34.424 INFO:teuthology.task.internal:roles: ubuntu@smithi183.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-03-27T08:33:34.424 INFO:teuthology.run_tasks:Running task console_log... 2024-03-27T08:33:34.464 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f373619bf70>, signals=[15]) 2024-03-27T08:33:34.464 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-27T08:33:34.469 INFO:teuthology.task.internal:Opening connections... 2024-03-27T08:33:34.470 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi183.front.sepia.ceph.com 2024-03-27T08:33:34.473 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T08:33:34.562 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-27T08:33:34.568 DEBUG:teuthology.orchestra.run.smithi183:> uname -m 2024-03-27T08:33:34.630 INFO:teuthology.orchestra.run.smithi183.stdout:x86_64 2024-03-27T08:33:34.630 DEBUG:teuthology.orchestra.run.smithi183:> cat /etc/os-release 2024-03-27T08:33:34.686 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="CentOS Stream" 2024-03-27T08:33:34.686 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="9" 2024-03-27T08:33:34.686 INFO:teuthology.orchestra.run.smithi183.stdout:ID="centos" 2024-03-27T08:33:34.686 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE="rhel fedora" 2024-03-27T08:33:34.686 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="9" 2024-03-27T08:33:34.686 INFO:teuthology.orchestra.run.smithi183.stdout:PLATFORM_ID="platform:el9" 2024-03-27T08:33:34.686 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-27T08:33:34.686 INFO:teuthology.orchestra.run.smithi183.stdout:ANSI_COLOR="0;31" 2024-03-27T08:33:34.686 INFO:teuthology.orchestra.run.smithi183.stdout:LOGO="fedora-logo-icon" 2024-03-27T08:33:34.686 INFO:teuthology.orchestra.run.smithi183.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-27T08:33:34.686 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://centos.org/" 2024-03-27T08:33:34.686 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-27T08:33:34.687 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-27T08:33:34.687 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-27T08:33:34.687 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2024-03-27T08:33:34.708 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-27T08:33:34.716 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-27T08:33:34.721 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-27T08:33:34.721 DEBUG:teuthology.orchestra.run.smithi183:> test '!' -e /home/ubuntu/cephtest 2024-03-27T08:33:34.741 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-27T08:33:34.747 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-27T08:33:34.747 DEBUG:teuthology.orchestra.run.smithi183:> test -z $(ls -A /var/lib/ceph) 2024-03-27T08:33:34.807 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-27T08:33:34.838 INFO:teuthology.run_tasks:Running task kernel... 2024-03-27T08:33:34.852 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-27T08:33:34.852 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-27T08:33:34.852 DEBUG:teuthology.orchestra.run.smithi183:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-27T08:33:34.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T08:33:34.867 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-03-27T08:33:34.922 INFO:teuthology.orchestra.run.smithi183.stdout:5.14.0-361.el9.x86_64 2024-03-27T08:33:34.922 INFO:teuthology.task.kernel:Running kernel on smithi183: 5.14.0-361.el9.x86_64 2024-03-27T08:33:34.922 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum install -y kernel 2024-03-27T08:33:36.930 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-27T08:33:38.580 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-03-27T08:33:40.454 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 5.3 MB/s | 8.0 MB 00:01 2024-03-27T08:33:42.413 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-03-27T08:33:46.472 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-27T08:33:47.990 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-03-27T08:33:50.257 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 9.6 MB/s | 21 MB 00:02 2024-03-27T08:33:55.132 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-27T08:33:57.297 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-27T08:33:57.297 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-27T08:33:57.392 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-27T08:33:57.401 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:33:57.402 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-03-27T08:33:57.402 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:33:57.402 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-27T08:33:57.402 INFO:teuthology.orchestra.run.smithi183.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-27T08:33:57.402 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-27T08:33:57.402 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-27T08:33:57.402 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-27T08:33:57.402 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-27T08:33:57.402 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:33:57.402 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-27T08:33:57.402 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:33:57.402 INFO:teuthology.orchestra.run.smithi183.stdout:Install 4 Packages 2024-03-27T08:33:57.402 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:33:57.403 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 76 M 2024-03-27T08:33:57.403 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 123 M 2024-03-27T08:33:57.403 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-27T08:33:57.718 INFO:teuthology.orchestra.run.smithi183.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 327 kB/s | 66 kB 00:00 2024-03-27T08:33:58.375 INFO:teuthology.orchestra.run.smithi183.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 17 MB/s | 15 MB 00:00 2024-03-27T08:33:59.200 INFO:teuthology.orchestra.run.smithi183.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 20 MB/s | 33 MB 00:01 2024-03-27T08:33:59.459 INFO:teuthology.orchestra.run.smithi183.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 16 MB/s | 27 MB 00:01 2024-03-27T08:33:59.462 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-27T08:33:59.462 INFO:teuthology.orchestra.run.smithi183.stdout:Total 37 MB/s | 76 MB 00:02 2024-03-27T08:34:00.127 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-27T08:34:00.480 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-27T08:34:00.480 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-27T08:34:01.472 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-27T08:34:01.473 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-27T08:34:03.942 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-27T08:34:04.552 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-27T08:34:04.666 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-27T08:34:07.415 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-27T08:34:07.553 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-27T08:34:12.276 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-27T08:34:12.461 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-27T08:34:17.219 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-27T08:34:46.700 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-27T08:34:46.700 INFO:teuthology.orchestra.run.smithi183.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-27T08:34:46.700 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:34:46.745 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-27T08:34:47.138 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-27T08:34:47.138 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-27T08:34:47.138 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-27T08:34:47.138 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-27T08:34:47.552 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-27T08:34:47.552 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:34:47.552 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-27T08:34:47.552 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-27T08:34:47.552 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-27T08:34:47.552 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-27T08:34:47.552 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-27T08:34:47.552 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:34:47.552 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-27T08:34:47.887 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2024-03-27T08:34:48.000 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-27T08:34:48.000 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-27T08:34:48.000 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-27T08:34:48.001 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-27T08:34:48.001 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-27T08:34:48.001 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-27T08:34:48.001 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi183.front.sepia.ceph.com, path=None, version=distro) 2024-03-27T08:34:48.001 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2024-03-27T08:34:48.080 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-27T08:34:48.081 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-27T08:34:48.081 DEBUG:teuthology.orchestra.run.smithi183:> sudo rpm -qi grub2-tools 2024-03-27T08:34:48.114 INFO:teuthology.orchestra.run.smithi183.stdout:Name : grub2-tools 2024-03-27T08:34:48.115 INFO:teuthology.orchestra.run.smithi183.stdout:Epoch : 1 2024-03-27T08:34:48.115 INFO:teuthology.orchestra.run.smithi183.stdout:Version : 2.06 2024-03-27T08:34:48.115 INFO:teuthology.orchestra.run.smithi183.stdout:Release : 68.el9 2024-03-27T08:34:48.115 INFO:teuthology.orchestra.run.smithi183.stdout:Architecture: x86_64 2024-03-27T08:34:48.115 INFO:teuthology.orchestra.run.smithi183.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-27T08:34:48.115 INFO:teuthology.orchestra.run.smithi183.stdout:Group : Unspecified 2024-03-27T08:34:48.115 INFO:teuthology.orchestra.run.smithi183.stdout:Size : 8264254 2024-03-27T08:34:48.115 INFO:teuthology.orchestra.run.smithi183.stdout:License : GPLv3+ 2024-03-27T08:34:48.115 INFO:teuthology.orchestra.run.smithi183.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-27T08:34:48.115 INFO:teuthology.orchestra.run.smithi183.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-27T08:34:48.115 INFO:teuthology.orchestra.run.smithi183.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-27T08:34:48.115 INFO:teuthology.orchestra.run.smithi183.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-27T08:34:48.116 INFO:teuthology.orchestra.run.smithi183.stdout:Packager : builder@centos.org 2024-03-27T08:34:48.116 INFO:teuthology.orchestra.run.smithi183.stdout:Vendor : CentOS 2024-03-27T08:34:48.116 INFO:teuthology.orchestra.run.smithi183.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-27T08:34:48.116 INFO:teuthology.orchestra.run.smithi183.stdout:Summary : Support tools for GRUB. 2024-03-27T08:34:48.116 INFO:teuthology.orchestra.run.smithi183.stdout:Description : 2024-03-27T08:34:48.116 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:34:48.116 INFO:teuthology.orchestra.run.smithi183.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-27T08:34:48.116 INFO:teuthology.orchestra.run.smithi183.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-27T08:34:48.116 INFO:teuthology.orchestra.run.smithi183.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-27T08:34:48.116 INFO:teuthology.orchestra.run.smithi183.stdout:hardware devices. 2024-03-27T08:34:48.116 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:34:48.116 INFO:teuthology.orchestra.run.smithi183.stdout:This subpackage provides tools for support of all platforms. 2024-03-27T08:34:48.117 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-27T08:34:48.117 INFO:teuthology.task.kernel:Updating grub on smithi183 to boot 5.14.0-432.el9.x86_64 2024-03-27T08:34:48.117 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-27T08:34:48.724 INFO:teuthology.orchestra.run.smithi183.stderr:Generating grub configuration file ... 2024-03-27T08:34:50.117 INFO:teuthology.orchestra.run.smithi183.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-27T08:34:50.150 INFO:teuthology.orchestra.run.smithi183.stderr:done 2024-03-27T08:34:50.153 DEBUG:teuthology.orchestra.run.smithi183:> mktemp 2024-03-27T08:34:50.168 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.Q5XFfpVwRS 2024-03-27T08:34:50.168 DEBUG:teuthology.orchestra.run.smithi183:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Q5XFfpVwRS 2024-03-27T08:34:50.237 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.Q5XFfpVwRS 2024-03-27T08:34:50.385 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.Q5XFfpVwRS is 6KB 2024-03-27T08:34:50.434 DEBUG:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.Q5XFfpVwRS 2024-03-27T08:34:50.448 DEBUG:teuthology.orchestra.run.smithi183:> sudo /bin/ls /boot/loader/entries || true 2024-03-27T08:34:50.512 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-27T08:34:50.512 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-27T08:34:50.513 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-27T08:34:50.513 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-27T08:34:50.514 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-27T08:34:50.665 DEBUG:teuthology.orchestra.run.smithi183:> sudo shutdown -r now 2024-03-27T08:35:20.697 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-27T08:35:20.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-27T08:35:20.699 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T08:36:20.759 DEBUG:teuthology.orchestra.remote:timed out 2024-03-27T08:36:29.765 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-27T08:36:29.767 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T08:36:30.348 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-03-27T08:36:30.739 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-27T08:36:30.739 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-27T08:36:30.739 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-03-27T08:36:30.753 INFO:teuthology.orchestra.run.smithi183.stdout:5.14.0-432.el9.x86_64 2024-03-27T08:36:30.753 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-27T08:36:30.753 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-27T08:36:30.753 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-27T08:36:31.754 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-27T08:36:31.754 DEBUG:teuthology.orchestra.run.smithi183:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-27T08:36:31.812 INFO:teuthology.orchestra.run.smithi183.stdout:ttyS1 2024-03-27T08:36:31.849 DEBUG:teuthology.parallel:result is None 2024-03-27T08:36:31.849 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-27T08:36:31.857 INFO:teuthology.task.internal:Creating test directory... 2024-03-27T08:36:31.857 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-27T08:36:31.871 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-27T08:36:31.878 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-27T08:36:31.907 INFO:teuthology.task.internal:Creating archive directory... 2024-03-27T08:36:31.907 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-27T08:36:31.931 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-27T08:36:31.937 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-27T08:36:31.937 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-27T08:36:32.003 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-27T08:36:32.016 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-27T08:36:32.018 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-27T08:36:32.025 INFO:teuthology.task.internal:Configuring sudo... 2024-03-27T08:36:32.025 DEBUG:teuthology.orchestra.run.smithi183:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-27T08:36:32.085 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-27T08:36:32.095 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-27T08:36:32.095 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-27T08:36:32.140 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-27T08:36:32.227 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-27T08:36:32.275 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-27T08:36:32.357 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-27T08:36:32.399 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-27T08:36:32.399 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-27T08:36:32.466 DEBUG:teuthology.orchestra.run.smithi183:> sudo service rsyslog restart 2024-03-27T08:36:32.550 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-27T08:36:32.701 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-27T08:36:32.709 INFO:teuthology.task.internal:Starting timer... 2024-03-27T08:36:32.709 INFO:teuthology.run_tasks:Running task pcp... 2024-03-27T08:36:32.719 INFO:teuthology.run_tasks:Running task selinux... 2024-03-27T08:36:32.728 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-03-27T08:36:32.728 DEBUG:teuthology.orchestra.run.smithi183:> sudo service auditd rotate 2024-03-27T08:36:32.793 INFO:teuthology.orchestra.run.smithi183.stdout:Rotating logs: 2024-03-27T08:36:32.795 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-27T08:36:32.795 DEBUG:teuthology.orchestra.run.smithi183:> /usr/sbin/getenforce 2024-03-27T08:36:32.818 INFO:teuthology.orchestra.run.smithi183.stdout:Permissive 2024-03-27T08:36:32.818 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi183.front.sepia.ceph.com': 'permissive'} 2024-03-27T08:36:32.818 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-27T08:36:32.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T08:36:32.886 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-27T08:36:32.886 DEBUG:teuthology.orchestra.run.smithi183:> sudo /usr/sbin/setenforce permissive 2024-03-27T08:36:32.962 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-27T08:36:32.972 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-27T08:36:32.989 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-27T08:36:32.991 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi183.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-27T08:40:48.546 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi183.front.sepia.ceph.com')] 2024-03-27T08:40:48.547 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-27T08:40:48.548 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T08:40:48.643 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-03-27T08:40:48.727 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-27T08:40:48.727 INFO:teuthology.run_tasks:Running task clock... 2024-03-27T08:40:48.794 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-27T08:40:48.794 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-27T08:40:48.794 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-27T08:40:48.826 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-27T08:40:48.842 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-27T08:40:48.871 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: ntpd: command not found 2024-03-27T08:40:48.885 INFO:teuthology.orchestra.run.smithi183.stdout:506 Cannot talk to daemon 2024-03-27T08:40:48.901 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-27T08:40:48.917 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-27T08:40:48.974 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-03-27T08:40:48.978 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-27T08:40:48.979 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-03-27T08:40:48.979 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T08:40:48.979 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T08:40:48.979 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T08:40:48.979 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T08:40:48.980 INFO:teuthology.run_tasks:Running task install... 2024-03-27T08:40:48.991 DEBUG:teuthology.task.install:project ceph 2024-03-27T08:40:48.991 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '022fc18aa97a30a2f3803bcc4f7d8f3544371a09'}} 2024-03-27T08:40:48.992 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '022fc18aa97a30a2f3803bcc4f7d8f3544371a09'} 2024-03-27T08:40:48.992 INFO:teuthology.task.install:Using flavor: crimson 2024-03-27T08:40:48.997 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-27T08:40:48.997 INFO:teuthology.task.install:extra packages: [] 2024-03-27T08:40:48.998 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '022fc18aa97a30a2f3803bcc4f7d8f3544371a09', 'tag': None, 'wait_for_package': False} 2024-03-27T08:40:48.998 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=022fc18aa97a30a2f3803bcc4f7d8f3544371a09 2024-03-27T08:40:49.154 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-matanb-crimson-foreign-ptr/022fc18aa97a30a2f3803bcc4f7d8f3544371a09/centos/9/flavors/crimson/ 2024-03-27T08:40:49.154 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2463.g022fc18a 2024-03-27T08:40:49.284 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-matanb-crimson-foreign-ptr/022fc18aa97a30a2f3803bcc4f7d8f3544371a09/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-matanb-crimson-foreign-ptr/022fc18aa97a30a2f3803bcc4f7d8f3544371a09/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-matanb-crimson-foreign-ptr/022fc18aa97a30a2f3803bcc4f7d8f3544371a09/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-27T08:40:49.285 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-27T08:40:49.285 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-27T08:40:49.314 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-27T08:40:49.314 DEBUG:teuthology.orchestra.run.smithi183:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/022fc18aa97a30a2f3803bcc4f7d8f3544371a09/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-27T08:40:49.382 DEBUG:teuthology.orchestra.run.smithi183:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-27T08:40:49.462 DEBUG:teuthology.orchestra.run.smithi183:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-27T08:40:49.529 INFO:teuthology.orchestra.run.smithi183.stdout:check_obsoletes = 1 2024-03-27T08:40:49.531 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum clean all 2024-03-27T08:40:49.773 INFO:teuthology.orchestra.run.smithi183.stdout:74 files removed 2024-03-27T08:40:49.801 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-radosgw 2024-03-27T08:40:50.626 INFO:teuthology.orchestra.run.smithi183.stdout:ceph packages for x86_64 165 kB/s | 87 kB 00:00 2024-03-27T08:40:51.084 INFO:teuthology.orchestra.run.smithi183.stdout:ceph noarch packages 28 kB/s | 11 kB 00:00 2024-03-27T08:40:51.544 INFO:teuthology.orchestra.run.smithi183.stdout:ceph source packages 4.9 kB/s | 2.0 kB 00:00 2024-03-27T08:40:51.962 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-27T08:40:53.375 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-27T08:41:02.727 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 914 kB/s | 8.0 MB 00:09 2024-03-27T08:41:04.503 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-03-27T08:41:08.462 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-27T08:41:10.089 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 16 kB 00:00 2024-03-27T08:41:11.717 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-27T08:41:16.672 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-03-27T08:41:19.151 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-27T08:41:19.165 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:41:19.165 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-27T08:41:19.165 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:41:19.165 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-27T08:41:19.165 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw x86_64 2:19.0.0-2463.g022fc18a.el9 ceph 9.6 M 2024-03-27T08:41:19.165 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrading: 2024-03-27T08:41:19.165 INFO:teuthology.orchestra.run.smithi183.stdout: librados2 x86_64 2:19.0.0-2463.g022fc18a.el9 ceph 3.4 M 2024-03-27T08:41:19.165 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1 x86_64 2:19.0.0-2463.g022fc18a.el9 ceph 3.7 M 2024-03-27T08:41:19.165 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-27T08:41:19.165 INFO:teuthology.orchestra.run.smithi183.stdout: boost-program-options 2024-03-27T08:41:19.165 INFO:teuthology.orchestra.run.smithi183.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-27T08:41:19.165 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base x86_64 2:19.0.0-2463.g022fc18a.el9 ceph 5.9 M 2024-03-27T08:41:19.165 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common x86_64 2:19.0.0-2463.g022fc18a.el9 ceph 19 M 2024-03-27T08:41:19.166 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux x86_64 2:19.0.0-2463.g022fc18a.el9 ceph 25 k 2024-03-27T08:41:19.166 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-27T08:41:19.166 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-27T08:41:19.166 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-27T08:41:19.166 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2 x86_64 2:19.0.0-2463.g022fc18a.el9 ceph 796 k 2024-03-27T08:41:19.166 INFO:teuthology.orchestra.run.smithi183.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-27T08:41:19.167 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-27T08:41:19.167 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-27T08:41:19.167 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1 x86_64 2:19.0.0-2463.g022fc18a.el9 ceph 548 k 2024-03-27T08:41:19.167 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-27T08:41:19.167 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2 x86_64 2:19.0.0-2463.g022fc18a.el9 ceph 5.0 M 2024-03-27T08:41:19.167 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-27T08:41:19.167 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-27T08:41:19.168 INFO:teuthology.orchestra.run.smithi183.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-27T08:41:19.168 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse 2024-03-27T08:41:19.168 INFO:teuthology.orchestra.run.smithi183.stdout: x86_64 2:19.0.0-2463.g022fc18a.el9 ceph 45 k 2024-03-27T08:41:19.168 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common x86_64 2:19.0.0-2463.g022fc18a.el9 ceph 134 k 2024-03-27T08:41:19.168 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs x86_64 2:19.0.0-2463.g022fc18a.el9 ceph 159 k 2024-03-27T08:41:19.169 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-27T08:41:19.169 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados x86_64 2:19.0.0-2463.g022fc18a.el9 ceph 315 k 2024-03-27T08:41:19.169 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd x86_64 2:19.0.0-2463.g022fc18a.el9 ceph 293 k 2024-03-27T08:41:19.169 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw x86_64 2:19.0.0-2463.g022fc18a.el9 ceph 99 k 2024-03-27T08:41:19.169 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-27T08:41:19.169 INFO:teuthology.orchestra.run.smithi183.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-27T08:41:19.170 INFO:teuthology.orchestra.run.smithi183.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-27T08:41:19.170 INFO:teuthology.orchestra.run.smithi183.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-27T08:41:19.170 INFO:teuthology.orchestra.run.smithi183.stdout:Installing weak dependencies: 2024-03-27T08:41:19.170 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-27T08:41:19.170 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:41:19.170 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-27T08:41:19.170 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:41:19.170 INFO:teuthology.orchestra.run.smithi183.stdout:Install 30 Packages 2024-03-27T08:41:19.171 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrade 2 Packages 2024-03-27T08:41:19.171 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:41:19.171 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 58 M 2024-03-27T08:41:19.171 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-27T08:41:23.973 INFO:teuthology.orchestra.run.smithi183.stdout:(1/32): ceph-base-19.0.0-2463.g022fc18a.el9.x86 1.3 MB/s | 5.9 MB 00:04 2024-03-27T08:41:24.065 INFO:teuthology.orchestra.run.smithi183.stdout:(2/32): ceph-selinux-19.0.0-2463.g022fc18a.el9. 275 kB/s | 25 kB 00:00 2024-03-27T08:41:24.565 INFO:teuthology.orchestra.run.smithi183.stdout:(3/32): libcephfs2-19.0.0-2463.g022fc18a.el9.x8 1.6 MB/s | 796 kB 00:00 2024-03-27T08:41:25.023 INFO:teuthology.orchestra.run.smithi183.stdout:(4/32): libradosstriper1-19.0.0-2463.g022fc18a. 1.2 MB/s | 548 kB 00:00 2024-03-27T08:41:27.049 INFO:teuthology.orchestra.run.smithi183.stdout:(5/32): librgw2-19.0.0-2463.g022fc18a.el9.x86_6 2.5 MB/s | 5.0 MB 00:02 2024-03-27T08:41:27.099 INFO:teuthology.orchestra.run.smithi183.stdout:(6/32): python3-ceph-argparse-19.0.0-2463.g022f 909 kB/s | 45 kB 00:00 2024-03-27T08:41:27.157 INFO:teuthology.orchestra.run.smithi183.stdout:(7/32): python3-ceph-common-19.0.0-2463.g022fc1 2.2 MB/s | 134 kB 00:00 2024-03-27T08:41:27.257 INFO:teuthology.orchestra.run.smithi183.stdout:(8/32): python3-cephfs-19.0.0-2463.g022fc18a.el 1.6 MB/s | 159 kB 00:00 2024-03-27T08:41:27.399 INFO:teuthology.orchestra.run.smithi183.stdout:(9/32): python3-rados-19.0.0-2463.g022fc18a.el9 2.2 MB/s | 315 kB 00:00 2024-03-27T08:41:27.500 INFO:teuthology.orchestra.run.smithi183.stdout:(10/32): python3-rbd-19.0.0-2463.g022fc18a.el9. 2.9 MB/s | 293 kB 00:00 2024-03-27T08:41:27.558 INFO:teuthology.orchestra.run.smithi183.stdout:(11/32): python3-rgw-19.0.0-2463.g022fc18a.el9. 1.7 MB/s | 99 kB 00:00 2024-03-27T08:41:27.667 INFO:teuthology.orchestra.run.smithi183.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 980 kB/s | 106 kB 00:00 2024-03-27T08:41:27.709 INFO:teuthology.orchestra.run.smithi183.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-27T08:41:27.742 INFO:teuthology.orchestra.run.smithi183.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-27T08:41:27.792 INFO:teuthology.orchestra.run.smithi183.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-27T08:41:27.834 INFO:teuthology.orchestra.run.smithi183.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-27T08:41:27.868 INFO:teuthology.orchestra.run.smithi183.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-27T08:41:28.093 INFO:teuthology.orchestra.run.smithi183.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 982 kB/s | 221 kB 00:00 2024-03-27T08:41:28.152 INFO:teuthology.orchestra.run.smithi183.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 618 kB/s | 36 kB 00:00 2024-03-27T08:41:28.210 INFO:teuthology.orchestra.run.smithi183.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 816 kB/s | 47 kB 00:00 2024-03-27T08:41:28.502 INFO:teuthology.orchestra.run.smithi183.stdout:(21/32): ceph-radosgw-19.0.0-2463.g022fc18a.el9 1.0 MB/s | 9.6 MB 00:09 2024-03-27T08:41:28.586 INFO:teuthology.orchestra.run.smithi183.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 824 kB/s | 309 kB 00:00 2024-03-27T08:41:28.653 INFO:teuthology.orchestra.run.smithi183.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 389 kB/s | 26 kB 00:00 2024-03-27T08:41:28.720 INFO:teuthology.orchestra.run.smithi183.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 731 kB/s | 49 kB 00:00 2024-03-27T08:41:28.787 INFO:teuthology.orchestra.run.smithi183.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-27T08:41:28.995 INFO:teuthology.orchestra.run.smithi183.stdout:(26/32): ceph-common-19.0.0-2463.g022fc18a.el9. 2.0 MB/s | 19 MB 00:09 2024-03-27T08:41:29.029 INFO:teuthology.orchestra.run.smithi183.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.4 MB/s | 838 kB 00:00 2024-03-27T08:41:29.163 INFO:teuthology.orchestra.run.smithi183.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-03-27T08:41:29.280 INFO:teuthology.orchestra.run.smithi183.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.7 MB/s | 4.4 MB 00:00 2024-03-27T08:41:29.313 INFO:teuthology.orchestra.run.smithi183.stdout:(30/32): re2-20211101-3.el9.x86_64.rpm 605 kB/s | 192 kB 00:00 2024-03-27T08:41:29.855 INFO:teuthology.orchestra.run.smithi183.stdout:(31/32): librbd1-19.0.0-2463.g022fc18a.el9.x86_ 6.4 MB/s | 3.7 MB 00:00 2024-03-27T08:41:30.163 INFO:teuthology.orchestra.run.smithi183.stdout:(32/32): librados2-19.0.0-2463.g022fc18a.el9.x8 3.4 MB/s | 3.4 MB 00:01 2024-03-27T08:41:30.167 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-27T08:41:30.167 INFO:teuthology.orchestra.run.smithi183.stdout:Total 5.3 MB/s | 58 MB 00:10 2024-03-27T08:41:30.200 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-27T08:41:30.240 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-27T08:41:30.240 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-27T08:41:30.710 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-27T08:41:30.711 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-27T08:41:32.165 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-27T08:41:32.602 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-27T08:41:32.665 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librados2-2:19.0.0-2463.g022fc18a.el9.x86_64 2/34 2024-03-27T08:41:32.740 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:19.0.0-2463.g022fc18a.el9.x86_64 2/34 2024-03-27T08:41:32.837 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rados-2:19.0.0-2463.g022fc18a.el9.x86_64 3/34 2024-03-27T08:41:32.866 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs2-2:19.0.0-2463.g022fc18a.el9.x86_64 4/34 2024-03-27T08:41:32.929 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephfs2-2:19.0.0-2463.g022fc18a.el9.x86_64 4/34 2024-03-27T08:41:32.967 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-03-27T08:41:32.995 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-03-27T08:41:33.021 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-03-27T08:41:33.044 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-03-27T08:41:33.059 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-03-27T08:41:33.211 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-03-27T08:41:33.243 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-03-27T08:41:33.592 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-03-27T08:41:33.640 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librbd1-2:19.0.0-2463.g022fc18a.el9.x86_64 11/34 2024-03-27T08:41:33.706 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:19.0.0-2463.g022fc18a.el9.x86_64 11/34 2024-03-27T08:41:33.748 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-argparse-2:19.0.0-2463.g022fc18a.el9. 12/34 2024-03-27T08:41:33.804 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cephfs-2:19.0.0-2463.g022fc18a.el9.x86_64 13/34 2024-03-27T08:41:33.837 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rbd-2:19.0.0-2463.g022fc18a.el9.x86_64 14/34 2024-03-27T08:41:33.925 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-03-27T08:41:33.966 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libradosstriper1-2:19.0.0-2463.g022fc18a.el9.x86_6 16/34 2024-03-27T08:41:34.043 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2463.g022fc18a.el9.x86_6 16/34 2024-03-27T08:41:34.082 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-03-27T08:41:34.194 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-03-27T08:41:34.225 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-03-27T08:41:34.267 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-03-27T08:41:34.298 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-03-27T08:41:34.326 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-03-27T08:41:34.358 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-03-27T08:41:34.399 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-27T08:41:34.949 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-27T08:41:34.972 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librgw2-2:19.0.0-2463.g022fc18a.el9.x86_64 26/34 2024-03-27T08:41:35.027 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librgw2-2:19.0.0-2463.g022fc18a.el9.x86_64 26/34 2024-03-27T08:41:35.087 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rgw-2:19.0.0-2463.g022fc18a.el9.x86_64 27/34 2024-03-27T08:41:35.116 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-common-2:19.0.0-2463.g022fc18a.el9.x8 28/34 2024-03-27T08:41:37.505 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:19.0.0-2463.g022fc18a.el9.x86_64 29/34 2024-03-27T08:41:37.538 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-common-2:19.0.0-2463.g022fc18a.el9.x86_64 29/34 2024-03-27T08:41:38.228 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:19.0.0-2463.g022fc18a.el9.x86_64 29/34 2024-03-27T08:41:38.307 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-base-2:19.0.0-2463.g022fc18a.el9.x86_64 30/34 2024-03-27T08:41:38.382 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-base-2:19.0.0-2463.g022fc18a.el9.x86_64 30/34 2024-03-27T08:41:38.382 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-27T08:41:38.382 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-27T08:41:38.382 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:41:38.425 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-selinux-2:19.0.0-2463.g022fc18a.el9.x86_64 31/34 2024-03-27T08:41:48.100 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2463.g022fc18a.el9.x86_64 31/34 2024-03-27T08:41:48.100 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /sys 2024-03-27T08:41:48.100 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /proc 2024-03-27T08:41:48.100 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /mnt 2024-03-27T08:41:48.100 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /var/tmp 2024-03-27T08:41:48.100 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /home 2024-03-27T08:41:48.100 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /root 2024-03-27T08:41:48.100 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /tmp 2024-03-27T08:41:48.100 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:41:49.105 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-radosgw-2:19.0.0-2463.g022fc18a.el9.x86_64 32/34 2024-03-27T08:41:49.130 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2463.g022fc18a.el9.x86_64 32/34 2024-03-27T08:41:49.130 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T08:41:49.130 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-27T08:41:49.130 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-27T08:41:49.130 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-27T08:41:49.130 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:41:49.132 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-27T08:41:49.199 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-27T08:41:49.199 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-27T08:41:50.426 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-27T08:41:50.426 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-base-2:19.0.0-2463.g022fc18a.el9.x86_64 1/34 2024-03-27T08:41:50.426 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-common-2:19.0.0-2463.g022fc18a.el9.x86_64 2/34 2024-03-27T08:41:50.426 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-radosgw-2:19.0.0-2463.g022fc18a.el9.x86_64 3/34 2024-03-27T08:41:50.426 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-selinux-2:19.0.0-2463.g022fc18a.el9.x86_64 4/34 2024-03-27T08:41:50.426 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs2-2:19.0.0-2463.g022fc18a.el9.x86_64 5/34 2024-03-27T08:41:50.426 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libradosstriper1-2:19.0.0-2463.g022fc18a.el9.x86_6 6/34 2024-03-27T08:41:50.427 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librgw2-2:19.0.0-2463.g022fc18a.el9.x86_64 7/34 2024-03-27T08:41:50.427 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2463.g022fc18a.el9. 8/34 2024-03-27T08:41:50.427 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-common-2:19.0.0-2463.g022fc18a.el9.x8 9/34 2024-03-27T08:41:50.427 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cephfs-2:19.0.0-2463.g022fc18a.el9.x86_64 10/34 2024-03-27T08:41:50.427 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rados-2:19.0.0-2463.g022fc18a.el9.x86_64 11/34 2024-03-27T08:41:50.427 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rbd-2:19.0.0-2463.g022fc18a.el9.x86_64 12/34 2024-03-27T08:41:50.427 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rgw-2:19.0.0-2463.g022fc18a.el9.x86_64 13/34 2024-03-27T08:41:50.427 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-27T08:41:50.427 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-27T08:41:50.427 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-27T08:41:50.427 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-27T08:41:50.427 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-27T08:41:50.427 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-27T08:41:50.427 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-27T08:41:50.427 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-27T08:41:50.428 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-27T08:41:50.428 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-27T08:41:50.428 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-27T08:41:50.428 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-27T08:41:50.428 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-27T08:41:50.428 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-27T08:41:50.428 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-27T08:41:50.428 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-27T08:41:50.428 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-27T08:41:50.429 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:19.0.0-2463.g022fc18a.el9.x86_64 31/34 2024-03-27T08:41:50.429 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-27T08:41:50.429 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:19.0.0-2463.g022fc18a.el9.x86_64 33/34 2024-03-27T08:41:51.557 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-27T08:41:51.557 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:41:51.557 INFO:teuthology.orchestra.run.smithi183.stdout:Upgraded: 2024-03-27T08:41:51.557 INFO:teuthology.orchestra.run.smithi183.stdout: librados2-2:19.0.0-2463.g022fc18a.el9.x86_64 2024-03-27T08:41:51.557 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1-2:19.0.0-2463.g022fc18a.el9.x86_64 2024-03-27T08:41:51.558 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-27T08:41:51.558 INFO:teuthology.orchestra.run.smithi183.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-27T08:41:51.558 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base-2:19.0.0-2463.g022fc18a.el9.x86_64 2024-03-27T08:41:51.558 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common-2:19.0.0-2463.g022fc18a.el9.x86_64 2024-03-27T08:41:51.558 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw-2:19.0.0-2463.g022fc18a.el9.x86_64 2024-03-27T08:41:51.558 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux-2:19.0.0-2463.g022fc18a.el9.x86_64 2024-03-27T08:41:51.558 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-27T08:41:51.558 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-27T08:41:51.558 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-27T08:41:51.558 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2-2:19.0.0-2463.g022fc18a.el9.x86_64 2024-03-27T08:41:51.558 INFO:teuthology.orchestra.run.smithi183.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-27T08:41:51.558 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-27T08:41:51.558 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-27T08:41:51.558 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1-2:19.0.0-2463.g022fc18a.el9.x86_64 2024-03-27T08:41:51.559 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-27T08:41:51.559 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2-2:19.0.0-2463.g022fc18a.el9.x86_64 2024-03-27T08:41:51.559 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-27T08:41:51.559 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-27T08:41:51.559 INFO:teuthology.orchestra.run.smithi183.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-27T08:41:51.559 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse-2:19.0.0-2463.g022fc18a.el9.x86_64 2024-03-27T08:41:51.559 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common-2:19.0.0-2463.g022fc18a.el9.x86_64 2024-03-27T08:41:51.559 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs-2:19.0.0-2463.g022fc18a.el9.x86_64 2024-03-27T08:41:51.559 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-27T08:41:51.559 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados-2:19.0.0-2463.g022fc18a.el9.x86_64 2024-03-27T08:41:51.559 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd-2:19.0.0-2463.g022fc18a.el9.x86_64 2024-03-27T08:41:51.559 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw-2:19.0.0-2463.g022fc18a.el9.x86_64 2024-03-27T08:41:51.559 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-27T08:41:51.559 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-27T08:41:51.559 INFO:teuthology.orchestra.run.smithi183.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-27T08:41:51.559 INFO:teuthology.orchestra.run.smithi183.stdout: re2-1:20211101-3.el9.x86_64 2024-03-27T08:41:51.560 INFO:teuthology.orchestra.run.smithi183.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-27T08:41:51.560 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:41:51.560 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-27T08:41:51.912 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-test 2024-03-27T08:41:52.486 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:36 ago on Wed 27 Mar 2024 08:41:16 AM UTC. 2024-03-27T08:41:52.637 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-27T08:41:52.637 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:41:52.638 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-27T08:41:52.638 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:41:52.638 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-27T08:41:52.638 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test x86_64 2:19.0.0-2463.g022fc18a.el9 ceph 87 M 2024-03-27T08:41:52.638 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-27T08:41:52.638 INFO:teuthology.orchestra.run.smithi183.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-27T08:41:52.639 INFO:teuthology.orchestra.run.smithi183.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-03-27T08:41:52.639 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite x86_64 2:19.0.0-2463.g022fc18a.el9 ceph 174 k 2024-03-27T08:41:52.639 INFO:teuthology.orchestra.run.smithi183.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-03-27T08:41:52.639 INFO:teuthology.orchestra.run.smithi183.stdout: lksctp-tools x86_64 1.0.19-2.el9 CentOS-BaseOS 90 k 2024-03-27T08:41:52.639 INFO:teuthology.orchestra.run.smithi183.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-27T08:41:52.639 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-27T08:41:52.639 INFO:teuthology.orchestra.run.smithi183.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-27T08:41:52.640 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-27T08:41:52.640 INFO:teuthology.orchestra.run.smithi183.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-03-27T08:41:52.640 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:41:52.640 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-27T08:41:52.640 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:41:52.640 INFO:teuthology.orchestra.run.smithi183.stdout:Install 11 Packages 2024-03-27T08:41:52.640 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:41:52.641 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 90 M 2024-03-27T08:41:52.642 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 819 M 2024-03-27T08:41:52.642 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-27T08:41:53.137 INFO:teuthology.orchestra.run.smithi183.stdout:(1/11): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-27T08:41:53.228 INFO:teuthology.orchestra.run.smithi183.stdout:(2/11): protobuf-3.14.0-13.el9.x86_64.rpm 11 MB/s | 1.0 MB 00:00 2024-03-27T08:41:53.270 INFO:teuthology.orchestra.run.smithi183.stdout:(3/11): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-27T08:41:53.304 INFO:teuthology.orchestra.run.smithi183.stdout:(4/11): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-27T08:41:53.329 INFO:teuthology.orchestra.run.smithi183.stdout:(5/11): lksctp-tools-1.0.19-2.el9.x86_64.rpm 3.5 MB/s | 90 kB 00:00 2024-03-27T08:41:53.396 INFO:teuthology.orchestra.run.smithi183.stdout:(6/11): libcephsqlite-19.0.0-2463.g022fc18a.el9 393 kB/s | 174 kB 00:00 2024-03-27T08:41:53.497 INFO:teuthology.orchestra.run.smithi183.stdout:(7/11): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-27T08:41:54.014 INFO:teuthology.orchestra.run.smithi183.stdout:(8/11): libubsan-11.4.1-3.el9.x86_64.rpm 362 kB/s | 187 kB 00:00 2024-03-27T08:41:54.114 INFO:teuthology.orchestra.run.smithi183.stdout:(9/11): libasan8-13.2.1-6.3.el9.x86_64.rpm 653 kB/s | 469 kB 00:00 2024-03-27T08:41:54.181 INFO:teuthology.orchestra.run.smithi183.stdout:(10/11): yaml-cpp-0.6.3-5.el9.x86_64.rpm 731 kB/s | 122 kB 00:00 2024-03-27T08:42:02.447 INFO:teuthology.orchestra.run.smithi183.stdout:(11/11): ceph-test-19.0.0-2463.g022fc18a.el9.x8 9.2 MB/s | 87 MB 00:09 2024-03-27T08:42:02.450 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-27T08:42:02.450 INFO:teuthology.orchestra.run.smithi183.stdout:Total 9.2 MB/s | 90 MB 00:09 2024-03-27T08:42:02.491 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-27T08:42:02.514 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-27T08:42:02.514 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-27T08:42:03.152 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-27T08:42:03.152 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-27T08:42:04.173 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-27T08:42:04.235 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/11 2024-03-27T08:42:04.274 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 2/11 2024-03-27T08:42:04.298 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-03-27T08:42:04.351 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-03-27T08:42:04.381 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 4/11 2024-03-27T08:42:04.459 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/11 2024-03-27T08:42:04.504 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/11 2024-03-27T08:42:04.527 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 7/11 2024-03-27T08:42:04.568 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 8/11 2024-03-27T08:42:04.610 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : jq-1.6-16.el9.x86_64 9/11 2024-03-27T08:42:04.637 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephsqlite-2:19.0.0-2463.g022fc18a.el9.x86_64 10/11 2024-03-27T08:42:16.163 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2463.g022fc18a.el9.x86_64 10/11 2024-03-27T08:42:16.236 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-test-2:19.0.0-2463.g022fc18a.el9.x86_64 11/11 2024-03-27T08:42:22.360 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-test-2:19.0.0-2463.g022fc18a.el9.x86_64 11/11 2024-03-27T08:42:22.360 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-test-2:19.0.0-2463.g022fc18a.el9.x86_64 1/11 2024-03-27T08:42:22.360 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephsqlite-2:19.0.0-2463.g022fc18a.el9.x86_64 2/11 2024-03-27T08:42:22.360 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/11 2024-03-27T08:42:22.360 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/11 2024-03-27T08:42:22.360 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/11 2024-03-27T08:42:22.360 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/11 2024-03-27T08:42:22.360 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 7/11 2024-03-27T08:42:22.360 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : jq-1.6-16.el9.x86_64 8/11 2024-03-27T08:42:22.361 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 9/11 2024-03-27T08:42:22.361 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 10/11 2024-03-27T08:42:22.764 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 11/11 2024-03-27T08:42:22.765 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:42:22.765 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-27T08:42:22.765 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test-2:19.0.0-2463.g022fc18a.el9.x86_64 2024-03-27T08:42:22.765 INFO:teuthology.orchestra.run.smithi183.stdout: jq-1.6-16.el9.x86_64 2024-03-27T08:42:22.765 INFO:teuthology.orchestra.run.smithi183.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-03-27T08:42:22.765 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite-2:19.0.0-2463.g022fc18a.el9.x86_64 2024-03-27T08:42:22.765 INFO:teuthology.orchestra.run.smithi183.stdout: libubsan-11.4.1-3.el9.x86_64 2024-03-27T08:42:22.765 INFO:teuthology.orchestra.run.smithi183.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-03-27T08:42:22.765 INFO:teuthology.orchestra.run.smithi183.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-27T08:42:22.765 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-27T08:42:22.765 INFO:teuthology.orchestra.run.smithi183.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-27T08:42:22.765 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-27T08:42:22.765 INFO:teuthology.orchestra.run.smithi183.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-03-27T08:42:22.766 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:42:22.766 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-27T08:42:23.102 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph 2024-03-27T08:42:23.665 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:07 ago on Wed 27 Mar 2024 08:41:16 AM UTC. 2024-03-27T08:42:23.817 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-27T08:42:23.819 INFO:teuthology.orchestra.run.smithi183.stdout:========================================================================================= 2024-03-27T08:42:23.819 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-27T08:42:23.819 INFO:teuthology.orchestra.run.smithi183.stdout:========================================================================================= 2024-03-27T08:42:23.819 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-27T08:42:23.819 INFO:teuthology.orchestra.run.smithi183.stdout: ceph x86_64 2:19.0.0-2463.g022fc18a.el9 ceph 6.5 k 2024-03-27T08:42:23.819 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-27T08:42:23.819 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds x86_64 2:19.0.0-2463.g022fc18a.el9 ceph 2.2 M 2024-03-27T08:42:23.820 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr x86_64 2:19.0.0-2463.g022fc18a.el9 ceph 1.6 M 2024-03-27T08:42:23.820 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2463.g022fc18a.el9 ceph-noarch 249 k 2024-03-27T08:42:23.820 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon x86_64 2:19.0.0-2463.g022fc18a.el9 ceph 5.0 M 2024-03-27T08:42:23.820 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd x86_64 2:19.0.0-2463.g022fc18a.el9 ceph 71 M 2024-03-27T08:42:23.820 INFO:teuthology.orchestra.run.smithi183.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-27T08:42:23.820 INFO:teuthology.orchestra.run.smithi183.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-27T08:42:23.820 INFO:teuthology.orchestra.run.smithi183.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-27T08:42:23.820 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-27T08:42:23.820 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-27T08:42:23.820 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-27T08:42:23.820 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-27T08:42:23.820 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-27T08:42:23.820 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-27T08:42:23.820 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-27T08:42:23.820 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-27T08:42:23.821 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-27T08:42:23.821 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-27T08:42:23.821 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-27T08:42:23.821 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-27T08:42:23.821 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-27T08:42:23.821 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-27T08:42:23.821 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-27T08:42:23.821 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-27T08:42:23.821 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-27T08:42:23.821 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-27T08:42:23.821 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-27T08:42:23.821 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-27T08:42:23.821 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-27T08:42:23.821 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-27T08:42:23.821 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-27T08:42:23.822 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-27T08:42:23.822 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-27T08:42:23.822 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-27T08:42:23.822 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-27T08:42:23.822 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:42:23.822 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-27T08:42:23.822 INFO:teuthology.orchestra.run.smithi183.stdout:========================================================================================= 2024-03-27T08:42:23.822 INFO:teuthology.orchestra.run.smithi183.stdout:Install 36 Packages 2024-03-27T08:42:23.822 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:42:23.823 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 84 M 2024-03-27T08:42:23.823 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 921 M 2024-03-27T08:42:23.823 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-27T08:42:24.646 INFO:teuthology.orchestra.run.smithi183.stdout:(1/36): ceph-19.0.0-2463.g022fc18a.el9.x86_64.r 34 kB/s | 6.5 kB 00:00 2024-03-27T08:42:24.996 INFO:teuthology.orchestra.run.smithi183.stdout:(2/36): ceph-mds-19.0.0-2463.g022fc18a.el9.x86_ 4.0 MB/s | 2.2 MB 00:00 2024-03-27T08:42:25.029 INFO:teuthology.orchestra.run.smithi183.stdout:(3/36): ceph-mgr-19.0.0-2463.g022fc18a.el9.x86_ 2.7 MB/s | 1.6 MB 00:00 2024-03-27T08:42:25.113 INFO:teuthology.orchestra.run.smithi183.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2463.g022f 2.9 MB/s | 249 kB 00:00 2024-03-27T08:42:25.272 INFO:teuthology.orchestra.run.smithi183.stdout:(5/36): ceph-mon-19.0.0-2463.g022fc18a.el9.x86_ 8.0 MB/s | 5.0 MB 00:00 2024-03-27T08:42:25.305 INFO:teuthology.orchestra.run.smithi183.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 973 kB/s | 187 kB 00:00 2024-03-27T08:42:25.330 INFO:teuthology.orchestra.run.smithi183.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-27T08:42:25.347 INFO:teuthology.orchestra.run.smithi183.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 762 kB/s | 32 kB 00:00 2024-03-27T08:42:25.364 INFO:teuthology.orchestra.run.smithi183.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-27T08:42:25.381 INFO:teuthology.orchestra.run.smithi183.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-27T08:42:25.406 INFO:teuthology.orchestra.run.smithi183.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-27T08:42:25.457 INFO:teuthology.orchestra.run.smithi183.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-27T08:42:25.482 INFO:teuthology.orchestra.run.smithi183.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-03-27T08:42:25.507 INFO:teuthology.orchestra.run.smithi183.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-27T08:42:25.533 INFO:teuthology.orchestra.run.smithi183.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-27T08:42:25.633 INFO:teuthology.orchestra.run.smithi183.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 223 kB/s | 22 kB 00:00 2024-03-27T08:42:25.725 INFO:teuthology.orchestra.run.smithi183.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 575 kB/s | 125 kB 00:00 2024-03-27T08:42:25.925 INFO:teuthology.orchestra.run.smithi183.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 518 kB/s | 151 kB 00:00 2024-03-27T08:42:25.950 INFO:teuthology.orchestra.run.smithi183.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 193 kB/s | 43 kB 00:00 2024-03-27T08:42:26.042 INFO:teuthology.orchestra.run.smithi183.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-03-27T08:42:26.109 INFO:teuthology.orchestra.run.smithi183.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 160 kB/s | 11 kB 00:00 2024-03-27T08:42:26.176 INFO:teuthology.orchestra.run.smithi183.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 265 kB/s | 18 kB 00:00 2024-03-27T08:42:26.202 INFO:teuthology.orchestra.run.smithi183.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-03-27T08:42:26.243 INFO:teuthology.orchestra.run.smithi183.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 346 kB/s | 23 kB 00:00 2024-03-27T08:42:26.269 INFO:teuthology.orchestra.run.smithi183.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 290 kB/s | 19 kB 00:00 2024-03-27T08:42:26.311 INFO:teuthology.orchestra.run.smithi183.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 293 kB/s | 20 kB 00:00 2024-03-27T08:42:26.344 INFO:teuthology.orchestra.run.smithi183.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 616 kB/s | 46 kB 00:00 2024-03-27T08:42:26.386 INFO:teuthology.orchestra.run.smithi183.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-27T08:42:26.437 INFO:teuthology.orchestra.run.smithi183.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-03-27T08:42:26.462 INFO:teuthology.orchestra.run.smithi183.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 217 kB/s | 16 kB 00:00 2024-03-27T08:42:26.512 INFO:teuthology.orchestra.run.smithi183.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-27T08:42:26.538 INFO:teuthology.orchestra.run.smithi183.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 476 kB/s | 36 kB 00:00 2024-03-27T08:42:26.605 INFO:teuthology.orchestra.run.smithi183.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-27T08:42:26.647 INFO:teuthology.orchestra.run.smithi183.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-03-27T08:42:26.680 INFO:teuthology.orchestra.run.smithi183.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 265 kB/s | 20 kB 00:00 2024-03-27T08:42:29.747 INFO:teuthology.orchestra.run.smithi183.stdout:(36/36): ceph-osd-19.0.0-2463.g022fc18a.el9.x86 15 MB/s | 71 MB 00:04 2024-03-27T08:42:29.751 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-27T08:42:29.751 INFO:teuthology.orchestra.run.smithi183.stdout:Total 14 MB/s | 84 MB 00:05 2024-03-27T08:42:29.776 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-27T08:42:29.820 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-27T08:42:29.820 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-27T08:42:30.547 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-27T08:42:30.547 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-27T08:42:32.172 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-27T08:42:32.216 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-27T08:42:32.251 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-27T08:42:32.285 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-27T08:42:32.355 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-27T08:42:32.400 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-27T08:42:32.433 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-27T08:42:32.533 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-27T08:42:32.619 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-27T08:42:32.660 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-27T08:42:32.733 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-27T08:42:32.786 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-27T08:42:32.824 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-27T08:42:32.877 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-27T08:42:32.937 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-27T08:42:33.121 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-27T08:42:33.170 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-27T08:42:33.201 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-27T08:42:33.235 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-27T08:42:33.265 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-27T08:42:33.333 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-27T08:42:33.367 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-27T08:42:33.401 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-27T08:42:33.441 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-27T08:42:33.476 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-27T08:42:33.511 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-27T08:42:33.599 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-27T08:42:33.641 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-27T08:42:33.713 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-27T08:42:33.858 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-27T08:42:33.982 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-27T08:42:34.168 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2463.g022fc18a.el9. 31/36 2024-03-27T08:42:34.204 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-2:19.0.0-2463.g022fc18a.el9.x86_64 32/36 2024-03-27T08:42:34.230 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2463.g022fc18a.el9.x86_64 32/36 2024-03-27T08:42:34.230 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T08:42:34.230 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-27T08:42:34.230 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-27T08:42:34.230 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-27T08:42:34.230 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:42:45.021 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-osd-2:19.0.0-2463.g022fc18a.el9.x86_64 33/36 2024-03-27T08:42:45.048 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-osd-2:19.0.0-2463.g022fc18a.el9.x86_64 33/36 2024-03-27T08:42:45.048 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T08:42:45.049 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-27T08:42:45.049 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-27T08:42:45.049 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-27T08:42:45.049 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:42:45.580 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mon-2:19.0.0-2463.g022fc18a.el9.x86_64 34/36 2024-03-27T08:42:45.603 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mon-2:19.0.0-2463.g022fc18a.el9.x86_64 34/36 2024-03-27T08:42:45.603 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T08:42:45.603 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-27T08:42:45.603 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-27T08:42:45.603 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-27T08:42:45.603 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:42:45.830 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mds-2:19.0.0-2463.g022fc18a.el9.x86_64 35/36 2024-03-27T08:42:45.853 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mds-2:19.0.0-2463.g022fc18a.el9.x86_64 35/36 2024-03-27T08:42:45.853 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T08:42:45.854 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-27T08:42:45.854 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-27T08:42:45.854 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-27T08:42:45.854 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:42:45.911 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-2:19.0.0-2463.g022fc18a.el9.x86_64 36/36 2024-03-27T08:42:52.551 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-2:19.0.0-2463.g022fc18a.el9.x86_64 36/36 2024-03-27T08:42:52.551 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-2:19.0.0-2463.g022fc18a.el9.x86_64 1/36 2024-03-27T08:42:52.551 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mds-2:19.0.0-2463.g022fc18a.el9.x86_64 2/36 2024-03-27T08:42:52.551 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-2:19.0.0-2463.g022fc18a.el9.x86_64 3/36 2024-03-27T08:42:52.551 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mon-2:19.0.0-2463.g022fc18a.el9.x86_64 4/36 2024-03-27T08:42:52.551 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-osd-2:19.0.0-2463.g022fc18a.el9.x86_64 5/36 2024-03-27T08:42:52.551 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2463.g022fc18a.el9. 6/36 2024-03-27T08:42:52.552 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-27T08:42:52.552 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-27T08:42:52.552 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-27T08:42:52.552 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-27T08:42:52.552 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-27T08:42:52.552 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-27T08:42:52.552 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-27T08:42:52.552 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-27T08:42:52.552 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-27T08:42:52.552 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-27T08:42:52.552 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-27T08:42:52.552 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-27T08:42:52.553 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-27T08:42:52.553 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-27T08:42:52.553 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-27T08:42:52.553 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-27T08:42:52.553 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-27T08:42:52.553 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-27T08:42:52.553 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-27T08:42:52.553 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-27T08:42:52.553 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-27T08:42:52.553 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-27T08:42:52.553 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-27T08:42:52.553 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-27T08:42:52.554 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-27T08:42:52.554 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-27T08:42:52.554 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-27T08:42:52.554 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-27T08:42:52.554 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-27T08:42:53.403 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-27T08:42:53.403 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:42:53.404 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-27T08:42:53.404 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-2:19.0.0-2463.g022fc18a.el9.x86_64 2024-03-27T08:42:53.404 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds-2:19.0.0-2463.g022fc18a.el9.x86_64 2024-03-27T08:42:53.404 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-2:19.0.0-2463.g022fc18a.el9.x86_64 2024-03-27T08:42:53.404 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core-2:19.0.0-2463.g022fc18a.el9.noarch 2024-03-27T08:42:53.404 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon-2:19.0.0-2463.g022fc18a.el9.x86_64 2024-03-27T08:42:53.404 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd-2:19.0.0-2463.g022fc18a.el9.x86_64 2024-03-27T08:42:53.404 INFO:teuthology.orchestra.run.smithi183.stdout: lua-5.4.4-4.el9.x86_64 2024-03-27T08:42:53.404 INFO:teuthology.orchestra.run.smithi183.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-27T08:42:53.404 INFO:teuthology.orchestra.run.smithi183.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-27T08:42:53.404 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-27T08:42:53.404 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-27T08:42:53.404 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-27T08:42:53.404 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-27T08:42:53.404 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-27T08:42:53.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-27T08:42:53.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-27T08:42:53.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-27T08:42:53.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-27T08:42:53.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-27T08:42:53.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-27T08:42:53.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-27T08:42:53.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-27T08:42:53.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-27T08:42:53.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-27T08:42:53.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply-3.11-14.el9.noarch 2024-03-27T08:42:53.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-27T08:42:53.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-27T08:42:53.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-27T08:42:53.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-27T08:42:53.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-27T08:42:53.406 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-27T08:42:53.406 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-27T08:42:53.406 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-27T08:42:53.406 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-27T08:42:53.406 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-27T08:42:53.406 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-27T08:42:53.406 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:42:53.406 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-27T08:42:53.665 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-base 2024-03-27T08:42:54.228 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:38 ago on Wed 27 Mar 2024 08:41:16 AM UTC. 2024-03-27T08:42:54.324 INFO:teuthology.orchestra.run.smithi183.stdout:Package ceph-base-2:19.0.0-2463.g022fc18a.el9.x86_64 is already installed. 2024-03-27T08:42:54.377 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-27T08:42:54.378 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-27T08:42:54.378 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-27T08:42:54.436 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install cephadm 2024-03-27T08:42:54.991 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:38 ago on Wed 27 Mar 2024 08:41:16 AM UTC. 2024-03-27T08:42:55.140 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-27T08:42:55.140 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:42:55.140 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-27T08:42:55.140 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:42:55.140 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-27T08:42:55.140 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm noarch 2:19.0.0-2463.g022fc18a.el9 ceph-noarch 762 k 2024-03-27T08:42:55.140 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:42:55.140 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-27T08:42:55.141 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:42:55.141 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-27T08:42:55.141 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:42:55.141 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 762 k 2024-03-27T08:42:55.141 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 768 k 2024-03-27T08:42:55.141 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-27T08:42:55.474 INFO:teuthology.orchestra.run.smithi183.stdout:cephadm-19.0.0-2463.g022fc18a.el9.noarch.rpm 2.2 MB/s | 762 kB 00:00 2024-03-27T08:42:55.474 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-27T08:42:55.474 INFO:teuthology.orchestra.run.smithi183.stdout:Total 2.2 MB/s | 762 kB 00:00 2024-03-27T08:42:55.475 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-27T08:42:55.480 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-27T08:42:55.480 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-27T08:42:55.485 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-27T08:42:55.486 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-27T08:42:55.560 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-27T08:42:56.047 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephadm-2:19.0.0-2463.g022fc18a.el9.noarch 1/1 2024-03-27T08:42:56.128 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephadm-2:19.0.0-2463.g022fc18a.el9.noarch 1/1 2024-03-27T08:42:56.493 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephadm-2:19.0.0-2463.g022fc18a.el9.noarch 1/1 2024-03-27T08:42:56.762 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephadm-2:19.0.0-2463.g022fc18a.el9.noarch 1/1 2024-03-27T08:42:56.763 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:42:56.763 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-27T08:42:56.763 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm-2:19.0.0-2463.g022fc18a.el9.noarch 2024-03-27T08:42:56.763 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:42:56.763 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-27T08:42:56.978 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-immutable-object-cache 2024-03-27T08:42:57.548 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:41 ago on Wed 27 Mar 2024 08:41:16 AM UTC. 2024-03-27T08:42:57.701 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-27T08:42:57.702 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:42:57.702 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-27T08:42:57.702 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:42:57.702 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-27T08:42:57.702 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2463.g022fc18a.el9 ceph 157 k 2024-03-27T08:42:57.702 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:42:57.702 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-27T08:42:57.702 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:42:57.703 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-27T08:42:57.703 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:42:57.703 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 157 k 2024-03-27T08:42:57.703 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 640 k 2024-03-27T08:42:57.703 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-27T08:42:58.047 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-immutable-object-cache-19.0.0-2463.g022fc1 458 kB/s | 157 kB 00:00 2024-03-27T08:42:58.047 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-27T08:42:58.048 INFO:teuthology.orchestra.run.smithi183.stdout:Total 456 kB/s | 157 kB 00:00 2024-03-27T08:42:58.048 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-27T08:42:58.057 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-27T08:42:58.057 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-27T08:42:58.110 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-27T08:42:58.110 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-27T08:42:58.251 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-27T08:42:58.306 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2463.g022fc18a. 1/1 2024-03-27T08:42:58.330 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2463.g022fc18a. 1/1 2024-03-27T08:42:58.330 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T08:42:58.330 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-27T08:42:58.330 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:42:59.325 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2463.g022fc18a. 1/1 2024-03-27T08:42:59.325 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:42:59.326 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-27T08:42:59.326 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache-2:19.0.0-2463.g022fc18a.el9.x86_64 2024-03-27T08:42:59.326 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:42:59.326 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-27T08:42:59.567 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr 2024-03-27T08:43:00.139 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:44 ago on Wed 27 Mar 2024 08:41:16 AM UTC. 2024-03-27T08:43:00.239 INFO:teuthology.orchestra.run.smithi183.stdout:Package ceph-mgr-2:19.0.0-2463.g022fc18a.el9.x86_64 is already installed. 2024-03-27T08:43:00.291 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-27T08:43:00.292 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-27T08:43:00.292 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-27T08:43:00.349 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-dashboard 2024-03-27T08:43:00.904 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:44 ago on Wed 27 Mar 2024 08:41:16 AM UTC. 2024-03-27T08:43:01.052 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-27T08:43:01.053 INFO:teuthology.orchestra.run.smithi183.stdout:====================================================================================== 2024-03-27T08:43:01.053 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-27T08:43:01.053 INFO:teuthology.orchestra.run.smithi183.stdout:====================================================================================== 2024-03-27T08:43:01.053 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-27T08:43:01.053 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2463.g022fc18a.el9 ceph-noarch 3.5 M 2024-03-27T08:43:01.053 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-27T08:43:01.054 INFO:teuthology.orchestra.run.smithi183.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-27T08:43:01.054 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2463.g022fc18a.el9 ceph-noarch 34 k 2024-03-27T08:43:01.054 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2463.g022fc18a.el9 ceph-noarch 16 k 2024-03-27T08:43:01.054 INFO:teuthology.orchestra.run.smithi183.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-27T08:43:01.054 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-27T08:43:01.054 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-27T08:43:01.054 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-27T08:43:01.054 INFO:teuthology.orchestra.run.smithi183.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-27T08:43:01.054 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-27T08:43:01.054 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-27T08:43:01.054 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:01.054 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-27T08:43:01.054 INFO:teuthology.orchestra.run.smithi183.stdout:====================================================================================== 2024-03-27T08:43:01.055 INFO:teuthology.orchestra.run.smithi183.stdout:Install 11 Packages 2024-03-27T08:43:01.055 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:01.055 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 7.6 M 2024-03-27T08:43:01.055 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 100 M 2024-03-27T08:43:01.055 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-27T08:43:01.443 INFO:teuthology.orchestra.run.smithi183.stdout:(1/11): ceph-grafana-dashboards-19.0.0-2463.g02 136 kB/s | 34 kB 00:00 2024-03-27T08:43:01.468 INFO:teuthology.orchestra.run.smithi183.stdout:(2/11): ceph-prometheus-alerts-19.0.0-2463.g022 60 kB/s | 16 kB 00:00 2024-03-27T08:43:01.568 INFO:teuthology.orchestra.run.smithi183.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 238 kB 00:00 2024-03-27T08:43:01.769 INFO:teuthology.orchestra.run.smithi183.stdout:(4/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.7 MB/s | 548 kB 00:00 2024-03-27T08:43:01.802 INFO:teuthology.orchestra.run.smithi183.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 585 kB/s | 19 kB 00:00 2024-03-27T08:43:01.836 INFO:teuthology.orchestra.run.smithi183.stdout:(6/11): protobuf-compiler-3.14.0-13.el9.x86_64. 2.4 MB/s | 863 kB 00:00 2024-03-27T08:43:01.978 INFO:teuthology.orchestra.run.smithi183.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.0 MB/s | 144 kB 00:00 2024-03-27T08:43:02.011 INFO:teuthology.orchestra.run.smithi183.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 923 kB/s | 31 kB 00:00 2024-03-27T08:43:02.079 INFO:teuthology.orchestra.run.smithi183.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-27T08:43:02.221 INFO:teuthology.orchestra.run.smithi183.stdout:(10/11): ceph-mgr-dashboard-19.0.0-2463.g022fc1 3.4 MB/s | 3.5 MB 00:01 2024-03-27T08:43:02.262 INFO:teuthology.orchestra.run.smithi183.stdout:(11/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.4 MB/s | 2.0 MB 00:00 2024-03-27T08:43:02.265 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-27T08:43:02.265 INFO:teuthology.orchestra.run.smithi183.stdout:Total 6.3 MB/s | 7.6 MB 00:01 2024-03-27T08:43:02.283 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-27T08:43:02.299 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-27T08:43:02.299 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-27T08:43:02.481 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-27T08:43:02.481 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-27T08:43:02.959 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-27T08:43:03.044 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-03-27T08:43:03.074 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-03-27T08:43:03.190 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-03-27T08:43:03.315 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-03-27T08:43:03.361 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-03-27T08:43:03.444 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-03-27T08:43:03.496 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-03-27T08:43:03.524 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-03-27T08:43:03.557 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2463.g022fc18a.el9 9/11 2024-03-27T08:43:04.608 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2463.g022fc18a.el 10/11 2024-03-27T08:43:04.648 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2463.g022fc18a.el9.noa 11/11 2024-03-27T08:43:05.929 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2463.g022fc18a.el9.noa 11/11 2024-03-27T08:43:05.929 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2463.g022fc18a.el 1/11 2024-03-27T08:43:05.929 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2463.g022fc18a.el9.noa 2/11 2024-03-27T08:43:05.929 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2463.g022fc18a.el9 3/11 2024-03-27T08:43:05.929 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-03-27T08:43:05.929 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-03-27T08:43:05.929 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-03-27T08:43:05.929 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-03-27T08:43:05.930 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-03-27T08:43:05.930 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-03-27T08:43:05.930 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-03-27T08:43:06.254 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-03-27T08:43:06.254 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:06.254 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-27T08:43:06.254 INFO:teuthology.orchestra.run.smithi183.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-27T08:43:06.254 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards-2:19.0.0-2463.g022fc18a.el9.noarch 2024-03-27T08:43:06.254 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard-2:19.0.0-2463.g022fc18a.el9.noarch 2024-03-27T08:43:06.255 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts-2:19.0.0-2463.g022fc18a.el9.noarch 2024-03-27T08:43:06.255 INFO:teuthology.orchestra.run.smithi183.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-27T08:43:06.255 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-27T08:43:06.255 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-27T08:43:06.255 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-27T08:43:06.255 INFO:teuthology.orchestra.run.smithi183.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-27T08:43:06.255 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-27T08:43:06.255 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-27T08:43:06.255 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:06.255 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-27T08:43:06.551 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-27T08:43:07.104 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:51 ago on Wed 27 Mar 2024 08:41:16 AM UTC. 2024-03-27T08:43:07.254 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-27T08:43:07.256 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2024-03-27T08:43:07.256 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-27T08:43:07.256 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2024-03-27T08:43:07.256 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-27T08:43:07.256 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2463.g022fc18a.el9 ceph-noarch 7.4 M 2024-03-27T08:43:07.256 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-27T08:43:07.256 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-27T08:43:07.256 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-27T08:43:07.256 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-27T08:43:07.256 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-27T08:43:07.256 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-27T08:43:07.257 INFO:teuthology.orchestra.run.smithi183.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-27T08:43:07.257 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-27T08:43:07.257 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-27T08:43:07.257 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-27T08:43:07.257 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-27T08:43:07.257 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-27T08:43:07.257 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:07.257 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-27T08:43:07.257 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2024-03-27T08:43:07.257 INFO:teuthology.orchestra.run.smithi183.stdout:Install 12 Packages 2024-03-27T08:43:07.257 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:07.258 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 37 M 2024-03-27T08:43:07.258 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 209 M 2024-03-27T08:43:07.258 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-27T08:43:07.720 INFO:teuthology.orchestra.run.smithi183.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 249 kB/s | 32 kB 00:00 2024-03-27T08:43:07.744 INFO:teuthology.orchestra.run.smithi183.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 727 kB/s | 17 kB 00:00 2024-03-27T08:43:07.778 INFO:teuthology.orchestra.run.smithi183.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-27T08:43:07.829 INFO:teuthology.orchestra.run.smithi183.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-03-27T08:43:07.987 INFO:teuthology.orchestra.run.smithi183.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2024-03-27T08:43:08.062 INFO:teuthology.orchestra.run.smithi183.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-03-27T08:43:08.088 INFO:teuthology.orchestra.run.smithi183.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-27T08:43:08.388 INFO:teuthology.orchestra.run.smithi183.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2024-03-27T08:43:08.422 INFO:teuthology.orchestra.run.smithi183.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.4 MB/s | 803 kB 00:00 2024-03-27T08:43:08.447 INFO:teuthology.orchestra.run.smithi183.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 3.3 MB/s | 195 kB 00:00 2024-03-27T08:43:08.522 INFO:teuthology.orchestra.run.smithi183.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.4 MB/s | 245 kB 00:00 2024-03-27T08:43:09.439 INFO:teuthology.orchestra.run.smithi183.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-2 4.0 MB/s | 7.4 MB 00:01 2024-03-27T08:43:09.442 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-27T08:43:09.442 INFO:teuthology.orchestra.run.smithi183.stdout:Total 17 MB/s | 37 MB 00:02 2024-03-27T08:43:09.486 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-27T08:43:09.511 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-27T08:43:09.511 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-27T08:43:09.869 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-27T08:43:09.869 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-27T08:43:10.569 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-27T08:43:10.637 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-27T08:43:10.668 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-27T08:43:10.762 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-27T08:43:10.805 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-27T08:43:10.984 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-27T08:43:11.021 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-27T08:43:11.099 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-27T08:43:11.748 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-27T08:43:11.870 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-27T08:43:13.088 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-27T08:43:14.118 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-27T08:43:14.145 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2463.g022fc 12/12 2024-03-27T08:43:16.594 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2463.g022fc 12/12 2024-03-27T08:43:16.594 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2463.g022fc 1/12 2024-03-27T08:43:16.594 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-27T08:43:16.594 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-27T08:43:16.594 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-27T08:43:16.594 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-27T08:43:16.595 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-27T08:43:16.595 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-27T08:43:16.595 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-27T08:43:16.595 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-27T08:43:16.595 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-27T08:43:16.595 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-27T08:43:17.060 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-27T08:43:17.061 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:17.061 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-27T08:43:17.061 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2463.g022fc18a.el9.noarch 2024-03-27T08:43:17.061 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-27T08:43:17.061 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-27T08:43:17.061 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-27T08:43:17.061 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-27T08:43:17.061 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-27T08:43:17.061 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-27T08:43:17.061 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-27T08:43:17.061 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-27T08:43:17.061 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-27T08:43:17.061 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-27T08:43:17.061 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-27T08:43:17.062 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:17.062 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-27T08:43:17.438 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-rook 2024-03-27T08:43:18.016 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:02 ago on Wed 27 Mar 2024 08:41:16 AM UTC. 2024-03-27T08:43:18.177 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-27T08:43:18.178 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-27T08:43:18.178 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-27T08:43:18.178 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-27T08:43:18.178 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-27T08:43:18.178 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook noarch 2:19.0.0-2463.g022fc18a.el9 ceph-noarch 49 k 2024-03-27T08:43:18.178 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-27T08:43:18.179 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-27T08:43:18.179 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-27T08:43:18.179 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-27T08:43:18.179 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-27T08:43:18.179 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-27T08:43:18.179 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-27T08:43:18.179 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-27T08:43:18.179 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-27T08:43:18.179 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-27T08:43:18.179 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-27T08:43:18.179 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-27T08:43:18.179 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-27T08:43:18.179 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:18.179 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-27T08:43:18.179 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-27T08:43:18.180 INFO:teuthology.orchestra.run.smithi183.stdout:Install 13 Packages 2024-03-27T08:43:18.180 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:18.180 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 2.0 M 2024-03-27T08:43:18.180 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 26 M 2024-03-27T08:43:18.180 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-27T08:43:18.421 INFO:teuthology.orchestra.run.smithi183.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 195 kB/s | 24 kB 00:00 2024-03-27T08:43:18.437 INFO:teuthology.orchestra.run.smithi183.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 127 kB/s | 18 kB 00:00 2024-03-27T08:43:18.488 INFO:teuthology.orchestra.run.smithi183.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-27T08:43:18.513 INFO:teuthology.orchestra.run.smithi183.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-27T08:43:18.538 INFO:teuthology.orchestra.run.smithi183.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-27T08:43:18.563 INFO:teuthology.orchestra.run.smithi183.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-03-27T08:43:18.589 INFO:teuthology.orchestra.run.smithi183.stdout:(7/13): ceph-mgr-rook-19.0.0-2463.g022fc18a.el9 168 kB/s | 49 kB 00:00 2024-03-27T08:43:18.981 INFO:teuthology.orchestra.run.smithi183.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 73 kB/s | 32 kB 00:00 2024-03-27T08:43:19.006 INFO:teuthology.orchestra.run.smithi183.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 34 kB/s | 15 kB 00:00 2024-03-27T08:43:19.256 INFO:teuthology.orchestra.run.smithi183.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 236 kB/s | 59 kB 00:00 2024-03-27T08:43:19.281 INFO:teuthology.orchestra.run.smithi183.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 293 kB/s | 203 kB 00:00 2024-03-27T08:43:19.457 INFO:teuthology.orchestra.run.smithi183.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 448 kB/s | 90 kB 00:00 2024-03-27T08:43:19.549 INFO:teuthology.orchestra.run.smithi183.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.8 MB/s | 1.0 MB 00:00 2024-03-27T08:43:19.553 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-27T08:43:19.553 INFO:teuthology.orchestra.run.smithi183.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2024-03-27T08:43:19.553 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-27T08:43:19.567 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-27T08:43:19.567 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-27T08:43:19.659 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-27T08:43:19.659 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-27T08:43:20.175 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-27T08:43:20.335 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-27T08:43:20.400 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-27T08:43:20.455 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-27T08:43:20.484 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-27T08:43:20.525 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-27T08:43:20.626 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-27T08:43:20.737 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-27T08:43:20.819 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-27T08:43:21.541 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-27T08:43:21.593 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-27T08:43:21.627 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-27T08:43:21.675 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-27T08:43:21.696 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-rook-2:19.0.0-2463.g022fc18a.el9.noarch 13/13 2024-03-27T08:43:22.469 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2463.g022fc18a.el9.noarch 13/13 2024-03-27T08:43:22.469 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2463.g022fc18a.el9.noarch 1/13 2024-03-27T08:43:22.470 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-27T08:43:22.470 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-27T08:43:22.470 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-27T08:43:22.470 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-27T08:43:22.470 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-27T08:43:22.470 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-27T08:43:22.470 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-27T08:43:22.470 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-27T08:43:22.470 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-27T08:43:22.470 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-27T08:43:22.470 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-27T08:43:22.821 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-27T08:43:22.821 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:22.821 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-27T08:43:22.821 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook-2:19.0.0-2463.g022fc18a.el9.noarch 2024-03-27T08:43:22.821 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-27T08:43:22.821 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-27T08:43:22.821 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-27T08:43:22.822 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-27T08:43:22.822 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-27T08:43:22.822 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-27T08:43:22.822 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-27T08:43:22.822 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-27T08:43:22.822 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-27T08:43:22.822 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-27T08:43:22.822 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-27T08:43:22.822 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-27T08:43:22.822 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:22.822 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-27T08:43:23.055 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-cephadm 2024-03-27T08:43:23.619 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:07 ago on Wed 27 Mar 2024 08:41:16 AM UTC. 2024-03-27T08:43:23.771 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-27T08:43:23.772 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-27T08:43:23.772 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-27T08:43:23.772 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-27T08:43:23.772 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-27T08:43:23.772 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2463.g022fc18a.el9 ceph-noarch 141 k 2024-03-27T08:43:23.772 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-27T08:43:23.772 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-27T08:43:23.772 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-27T08:43:23.772 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-27T08:43:23.772 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-27T08:43:23.772 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-27T08:43:23.773 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:23.773 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-27T08:43:23.773 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-27T08:43:23.773 INFO:teuthology.orchestra.run.smithi183.stdout:Install 6 Packages 2024-03-27T08:43:23.773 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:23.773 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 6.8 M 2024-03-27T08:43:23.773 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 33 M 2024-03-27T08:43:23.774 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-27T08:43:24.203 INFO:teuthology.orchestra.run.smithi183.stdout:(1/6): ceph-mgr-cephadm-19.0.0-2463.g022fc18a.e 654 kB/s | 141 kB 00:00 2024-03-27T08:43:24.278 INFO:teuthology.orchestra.run.smithi183.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-27T08:43:24.520 INFO:teuthology.orchestra.run.smithi183.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.7 MB/s | 548 kB 00:00 2024-03-27T08:43:24.545 INFO:teuthology.orchestra.run.smithi183.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 237 kB/s | 58 kB 00:00 2024-03-27T08:43:24.562 INFO:teuthology.orchestra.run.smithi183.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.2 MB/s | 51 kB 00:00 2024-03-27T08:43:24.954 INFO:teuthology.orchestra.run.smithi183.stdout:(6/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 256 kB/s | 248 kB 00:00 2024-03-27T08:43:24.957 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-27T08:43:24.958 INFO:teuthology.orchestra.run.smithi183.stdout:Total 5.7 MB/s | 6.8 MB 00:01 2024-03-27T08:43:24.970 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-27T08:43:24.981 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-27T08:43:24.981 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-27T08:43:25.072 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-27T08:43:25.072 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-27T08:43:25.371 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-27T08:43:25.527 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-27T08:43:25.580 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-27T08:43:26.105 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-27T08:43:26.191 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-27T08:43:26.254 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-27T08:43:26.284 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2463.g022fc18a.el9.noarch 6/6 2024-03-27T08:43:27.099 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2463.g022fc18a.el9.noarch 6/6 2024-03-27T08:43:27.099 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2463.g022fc18a.el9.noarch 1/6 2024-03-27T08:43:27.099 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-27T08:43:27.099 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-27T08:43:27.099 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-27T08:43:27.099 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-27T08:43:27.391 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-27T08:43:27.391 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:27.391 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-27T08:43:27.391 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm-2:19.0.0-2463.g022fc18a.el9.noarch 2024-03-27T08:43:27.391 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-27T08:43:27.391 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-27T08:43:27.391 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-27T08:43:27.391 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-27T08:43:27.391 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-27T08:43:27.392 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:27.392 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-27T08:43:27.685 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-fuse 2024-03-27T08:43:28.248 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:12 ago on Wed 27 Mar 2024 08:41:16 AM UTC. 2024-03-27T08:43:28.399 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-27T08:43:28.399 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:43:28.399 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-27T08:43:28.399 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:43:28.399 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-27T08:43:28.399 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse x86_64 2:19.0.0-2463.g022fc18a.el9 ceph 870 k 2024-03-27T08:43:28.399 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-27T08:43:28.399 INFO:teuthology.orchestra.run.smithi183.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-27T08:43:28.399 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:28.400 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-27T08:43:28.400 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:43:28.400 INFO:teuthology.orchestra.run.smithi183.stdout:Install 2 Packages 2024-03-27T08:43:28.400 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:28.400 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 949 k 2024-03-27T08:43:28.400 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 4.0 M 2024-03-27T08:43:28.400 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-27T08:43:28.531 INFO:teuthology.orchestra.run.smithi183.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 602 kB/s | 79 kB 00:00 2024-03-27T08:43:28.948 INFO:teuthology.orchestra.run.smithi183.stdout:(2/2): ceph-fuse-19.0.0-2463.g022fc18a.el9.x86_ 1.6 MB/s | 870 kB 00:00 2024-03-27T08:43:28.949 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-27T08:43:28.949 INFO:teuthology.orchestra.run.smithi183.stdout:Total 1.7 MB/s | 949 kB 00:00 2024-03-27T08:43:28.950 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-27T08:43:28.972 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-27T08:43:28.972 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-27T08:43:29.044 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-27T08:43:29.045 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-27T08:43:29.238 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-27T08:43:29.383 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-27T08:43:29.457 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-fuse-2:19.0.0-2463.g022fc18a.el9.x86_64 2/2 2024-03-27T08:43:30.174 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2463.g022fc18a.el9.x86_64 2/2 2024-03-27T08:43:30.174 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-fuse-2:19.0.0-2463.g022fc18a.el9.x86_64 1/2 2024-03-27T08:43:30.486 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-27T08:43:30.486 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:30.486 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-27T08:43:30.486 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse-2:19.0.0-2463.g022fc18a.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-27T08:43:30.486 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:30.486 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-27T08:43:30.766 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-volume 2024-03-27T08:43:31.338 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:15 ago on Wed 27 Mar 2024 08:41:16 AM UTC. 2024-03-27T08:43:31.487 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-27T08:43:31.487 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:43:31.487 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-27T08:43:31.488 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:43:31.488 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-27T08:43:31.488 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume noarch 2:19.0.0-2463.g022fc18a.el9 ceph-noarch 264 k 2024-03-27T08:43:31.488 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-27T08:43:31.488 INFO:teuthology.orchestra.run.smithi183.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-27T08:43:31.488 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:31.488 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-27T08:43:31.488 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:43:31.488 INFO:teuthology.orchestra.run.smithi183.stdout:Install 2 Packages 2024-03-27T08:43:31.488 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:31.489 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 332 k 2024-03-27T08:43:31.489 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 1.5 M 2024-03-27T08:43:31.489 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-27T08:43:31.619 INFO:teuthology.orchestra.run.smithi183.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 527 kB/s | 69 kB 00:00 2024-03-27T08:43:31.986 INFO:teuthology.orchestra.run.smithi183.stdout:(2/2): ceph-volume-19.0.0-2463.g022fc18a.el9.no 531 kB/s | 264 kB 00:00 2024-03-27T08:43:31.987 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-27T08:43:31.988 INFO:teuthology.orchestra.run.smithi183.stdout:Total 666 kB/s | 332 kB 00:00 2024-03-27T08:43:31.988 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-27T08:43:31.997 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-27T08:43:31.997 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-27T08:43:32.040 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-27T08:43:32.040 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-27T08:43:32.255 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-27T08:43:32.433 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-27T08:43:32.481 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-volume-2:19.0.0-2463.g022fc18a.el9.noarch 2/2 2024-03-27T08:43:32.496 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-volume-2:19.0.0-2463.g022fc18a.el9.noarch 2/2 2024-03-27T08:43:32.496 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T08:43:32.496 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-27T08:43:32.496 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:33.270 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-volume-2:19.0.0-2463.g022fc18a.el9.noarch 1/2 2024-03-27T08:43:33.589 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-27T08:43:33.589 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:33.589 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-27T08:43:33.589 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume-2:19.0.0-2463.g022fc18a.el9.noarch 2024-03-27T08:43:33.589 INFO:teuthology.orchestra.run.smithi183.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-27T08:43:33.589 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:33.589 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-27T08:43:33.892 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librados-devel 2024-03-27T08:43:34.472 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:18 ago on Wed 27 Mar 2024 08:41:16 AM UTC. 2024-03-27T08:43:34.621 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-27T08:43:34.622 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:43:34.622 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-27T08:43:34.622 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:43:34.622 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-27T08:43:34.622 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel x86_64 2:19.0.0-2463.g022fc18a.el9 ceph 125 k 2024-03-27T08:43:34.622 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:34.622 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-27T08:43:34.622 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:43:34.622 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-27T08:43:34.622 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:34.623 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 125 k 2024-03-27T08:43:34.623 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 445 k 2024-03-27T08:43:34.623 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-27T08:43:35.175 INFO:teuthology.orchestra.run.smithi183.stdout:librados-devel-19.0.0-2463.g022fc18a.el9.x86_64 226 kB/s | 125 kB 00:00 2024-03-27T08:43:35.176 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-27T08:43:35.177 INFO:teuthology.orchestra.run.smithi183.stdout:Total 226 kB/s | 125 kB 00:00 2024-03-27T08:43:35.177 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-27T08:43:35.182 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-27T08:43:35.182 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-27T08:43:35.239 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-27T08:43:35.239 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-27T08:43:35.386 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-27T08:43:35.499 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librados-devel-2:19.0.0-2463.g022fc18a.el9.x86_64 1/1 2024-03-27T08:43:35.952 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados-devel-2:19.0.0-2463.g022fc18a.el9.x86_64 1/1 2024-03-27T08:43:36.234 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados-devel-2:19.0.0-2463.g022fc18a.el9.x86_64 1/1 2024-03-27T08:43:36.235 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:36.235 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-27T08:43:36.235 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel-2:19.0.0-2463.g022fc18a.el9.x86_64 2024-03-27T08:43:36.235 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:36.235 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-27T08:43:36.513 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install libcephfs2 2024-03-27T08:43:37.089 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:21 ago on Wed 27 Mar 2024 08:41:16 AM UTC. 2024-03-27T08:43:37.190 INFO:teuthology.orchestra.run.smithi183.stdout:Package libcephfs2-2:19.0.0-2463.g022fc18a.el9.x86_64 is already installed. 2024-03-27T08:43:37.243 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-27T08:43:37.244 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-27T08:43:37.244 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-27T08:43:37.316 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install libcephfs-devel 2024-03-27T08:43:37.885 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:21 ago on Wed 27 Mar 2024 08:41:16 AM UTC. 2024-03-27T08:43:38.032 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-27T08:43:38.033 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:43:38.033 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-27T08:43:38.033 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:43:38.033 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-27T08:43:38.033 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel x86_64 2:19.0.0-2463.g022fc18a.el9 ceph 32 k 2024-03-27T08:43:38.033 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:38.033 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-27T08:43:38.033 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:43:38.034 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-27T08:43:38.034 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:38.034 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 32 k 2024-03-27T08:43:38.034 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 142 k 2024-03-27T08:43:38.034 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-27T08:43:38.521 INFO:teuthology.orchestra.run.smithi183.stdout:libcephfs-devel-19.0.0-2463.g022fc18a.el9.x86_6 65 kB/s | 32 kB 00:00 2024-03-27T08:43:38.522 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-27T08:43:38.522 INFO:teuthology.orchestra.run.smithi183.stdout:Total 65 kB/s | 32 kB 00:00 2024-03-27T08:43:38.522 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-27T08:43:38.525 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-27T08:43:38.525 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-27T08:43:38.553 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-27T08:43:38.553 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-27T08:43:38.646 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-27T08:43:38.762 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs-devel-2:19.0.0-2463.g022fc18a.el9.x86_64 1/1 2024-03-27T08:43:39.160 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2463.g022fc18a.el9.x86_64 1/1 2024-03-27T08:43:39.409 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs-devel-2:19.0.0-2463.g022fc18a.el9.x86_64 1/1 2024-03-27T08:43:39.409 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:39.409 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-27T08:43:39.410 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel-2:19.0.0-2463.g022fc18a.el9.x86_64 2024-03-27T08:43:39.410 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:39.410 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-27T08:43:39.598 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librados2 2024-03-27T08:43:40.157 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:24 ago on Wed 27 Mar 2024 08:41:16 AM UTC. 2024-03-27T08:43:40.254 INFO:teuthology.orchestra.run.smithi183.stdout:Package librados2-2:19.0.0-2463.g022fc18a.el9.x86_64 is already installed. 2024-03-27T08:43:40.305 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-27T08:43:40.306 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-27T08:43:40.307 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-27T08:43:40.367 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librbd1 2024-03-27T08:43:40.927 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:24 ago on Wed 27 Mar 2024 08:41:16 AM UTC. 2024-03-27T08:43:41.024 INFO:teuthology.orchestra.run.smithi183.stdout:Package librbd1-2:19.0.0-2463.g022fc18a.el9.x86_64 is already installed. 2024-03-27T08:43:41.076 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-27T08:43:41.077 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-27T08:43:41.077 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-27T08:43:41.150 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rados 2024-03-27T08:43:41.710 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:25 ago on Wed 27 Mar 2024 08:41:16 AM UTC. 2024-03-27T08:43:41.807 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rados-2:19.0.0-2463.g022fc18a.el9.x86_64 is already installed. 2024-03-27T08:43:41.860 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-27T08:43:41.861 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-27T08:43:41.862 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-27T08:43:41.930 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rgw 2024-03-27T08:43:42.496 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:26 ago on Wed 27 Mar 2024 08:41:16 AM UTC. 2024-03-27T08:43:42.592 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rgw-2:19.0.0-2463.g022fc18a.el9.x86_64 is already installed. 2024-03-27T08:43:42.645 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-27T08:43:42.646 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-27T08:43:42.646 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-27T08:43:42.711 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-cephfs 2024-03-27T08:43:43.273 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:27 ago on Wed 27 Mar 2024 08:41:16 AM UTC. 2024-03-27T08:43:43.369 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-cephfs-2:19.0.0-2463.g022fc18a.el9.x86_64 is already installed. 2024-03-27T08:43:43.421 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-27T08:43:43.422 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-27T08:43:43.422 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-27T08:43:43.495 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rbd 2024-03-27T08:43:44.056 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:28 ago on Wed 27 Mar 2024 08:41:16 AM UTC. 2024-03-27T08:43:44.151 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rbd-2:19.0.0-2463.g022fc18a.el9.x86_64 is already installed. 2024-03-27T08:43:44.205 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-27T08:43:44.206 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-27T08:43:44.206 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-27T08:43:44.269 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-fuse 2024-03-27T08:43:44.842 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:28 ago on Wed 27 Mar 2024 08:41:16 AM UTC. 2024-03-27T08:43:44.991 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-27T08:43:44.992 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:43:44.992 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-03-27T08:43:44.992 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:43:44.992 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-27T08:43:44.992 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse x86_64 2:19.0.0-2463.g022fc18a.el9 ceph 90 k 2024-03-27T08:43:44.992 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:44.992 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-27T08:43:44.992 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:43:44.992 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-27T08:43:44.993 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:44.993 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 90 k 2024-03-27T08:43:44.993 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 313 k 2024-03-27T08:43:44.993 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-27T08:43:45.191 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-fuse-19.0.0-2463.g022fc18a.el9.x86_64.rpm 453 kB/s | 90 kB 00:00 2024-03-27T08:43:45.191 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-27T08:43:45.191 INFO:teuthology.orchestra.run.smithi183.stdout:Total 450 kB/s | 90 kB 00:00 2024-03-27T08:43:45.192 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-27T08:43:45.200 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-27T08:43:45.200 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-27T08:43:45.256 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-27T08:43:45.256 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-27T08:43:45.398 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-27T08:43:45.498 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-fuse-2:19.0.0-2463.g022fc18a.el9.x86_64 1/1 2024-03-27T08:43:45.881 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2463.g022fc18a.el9.x86_64 1/1 2024-03-27T08:43:46.184 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-fuse-2:19.0.0-2463.g022fc18a.el9.x86_64 1/1 2024-03-27T08:43:46.184 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:46.184 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-27T08:43:46.184 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse-2:19.0.0-2463.g022fc18a.el9.x86_64 2024-03-27T08:43:46.184 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:46.184 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-27T08:43:46.409 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-mirror 2024-03-27T08:43:46.981 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:30 ago on Wed 27 Mar 2024 08:41:16 AM UTC. 2024-03-27T08:43:47.136 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-27T08:43:47.137 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:43:47.137 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-27T08:43:47.137 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:43:47.137 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-27T08:43:47.137 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror x86_64 2:19.0.0-2463.g022fc18a.el9 ceph 3.4 M 2024-03-27T08:43:47.137 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:47.137 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-27T08:43:47.137 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:43:47.137 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-27T08:43:47.137 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:47.138 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 3.4 M 2024-03-27T08:43:47.138 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 29 M 2024-03-27T08:43:47.138 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-27T08:43:49.153 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-mirror-19.0.0-2463.g022fc18a.el9.x86_64.rpm 1.7 MB/s | 3.4 MB 00:02 2024-03-27T08:43:49.154 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-27T08:43:49.154 INFO:teuthology.orchestra.run.smithi183.stdout:Total 1.7 MB/s | 3.4 MB 00:02 2024-03-27T08:43:49.154 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-27T08:43:49.165 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-27T08:43:49.165 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-27T08:43:49.240 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-27T08:43:49.241 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-27T08:43:49.800 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-27T08:43:49.862 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-mirror-2:19.0.0-2463.g022fc18a.el9.x86_64 1/1 2024-03-27T08:43:49.887 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2463.g022fc18a.el9.x86_64 1/1 2024-03-27T08:43:49.887 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T08:43:49.887 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-27T08:43:49.887 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-27T08:43:49.887 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-27T08:43:49.887 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:51.112 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-mirror-2:19.0.0-2463.g022fc18a.el9.x86_64 1/1 2024-03-27T08:43:51.112 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:51.112 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-27T08:43:51.112 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror-2:19.0.0-2463.g022fc18a.el9.x86_64 2024-03-27T08:43:51.112 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:51.112 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-27T08:43:51.373 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-nbd 2024-03-27T08:43:51.946 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:35 ago on Wed 27 Mar 2024 08:41:16 AM UTC. 2024-03-27T08:43:52.095 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-27T08:43:52.096 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:43:52.096 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-03-27T08:43:52.096 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:43:52.096 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-27T08:43:52.096 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd x86_64 2:19.0.0-2463.g022fc18a.el9 ceph 175 k 2024-03-27T08:43:52.097 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:52.097 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-27T08:43:52.097 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-27T08:43:52.097 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-27T08:43:52.097 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:52.097 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 175 k 2024-03-27T08:43:52.097 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 673 k 2024-03-27T08:43:52.097 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-27T08:43:52.590 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-nbd-19.0.0-2463.g022fc18a.el9.x86_64.rpm 356 kB/s | 175 kB 00:00 2024-03-27T08:43:52.590 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-27T08:43:52.590 INFO:teuthology.orchestra.run.smithi183.stdout:Total 355 kB/s | 175 kB 00:00 2024-03-27T08:43:52.591 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-27T08:43:52.601 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-27T08:43:52.601 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-27T08:43:52.658 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-27T08:43:52.658 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-27T08:43:52.816 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-27T08:43:52.938 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-nbd-2:19.0.0-2463.g022fc18a.el9.x86_64 1/1 2024-03-27T08:43:53.378 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2463.g022fc18a.el9.x86_64 1/1 2024-03-27T08:43:53.694 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-nbd-2:19.0.0-2463.g022fc18a.el9.x86_64 1/1 2024-03-27T08:43:53.695 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:53.695 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-27T08:43:53.695 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd-2:19.0.0-2463.g022fc18a.el9.x86_64 2024-03-27T08:43:53.695 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:53.695 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-27T08:43:53.945 DEBUG:teuthology.parallel:result is None 2024-03-27T08:43:53.946 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=022fc18aa97a30a2f3803bcc4f7d8f3544371a09 2024-03-27T08:43:54.169 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-27T08:43:54.190 INFO:teuthology.orchestra.run.smithi183.stdout:19.0.0-2463.g022fc18a.el9 2024-03-27T08:43:54.191 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2463.g022fc18a.el9 2024-03-27T08:43:54.191 INFO:teuthology.task.install:The correct ceph version 19.0.0-2463.g022fc18a is installed. 2024-03-27T08:43:54.193 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-27T08:43:54.193 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-27T08:43:54.193 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-27T08:43:54.259 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-27T08:43:54.260 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-27T08:43:54.260 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/daemon-helper 2024-03-27T08:43:54.327 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-27T08:43:54.393 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-27T08:43:54.394 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-27T08:43:54.394 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-27T08:43:54.463 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-27T08:43:54.529 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-27T08:43:54.530 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-27T08:43:54.530 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/stdin-killer 2024-03-27T08:43:54.596 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-27T08:43:54.669 INFO:teuthology.run_tasks:Running task ceph... 2024-03-27T08:43:54.788 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-27T08:43:54.788 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 777 /var/log/ceph 2024-03-27T08:43:54.815 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-27T08:43:54.815 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-27T08:43:54.883 INFO:tasks.ceph:Creating extra log directories... 2024-03-27T08:43:54.883 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-27T08:43:54.957 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-27T08:43:54.957 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-27T08:43:54.958 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2024-03-27_08:05:55-crimson-rados-wip-matanb-crimson-foreign-ptr-distro-crimson-smithi/7625904', 'branch': 'wip-matanb-crimson-foreign-ptr', 'description': 'crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_lock_and_fence}', 'email': None, 'first_in_suite': False, 'job_id': '7625904', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2024-03-27_08:05:55-crimson-rados-wip-matanb-crimson-foreign-ptr-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-matanb-crimson-foreign-ptr'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '022fc18aa97a30a2f3803bcc4f7d8f3544371a09'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '022fc18aa97a30a2f3803bcc4f7d8f3544371a09'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'eae914f097640d21427554e7f6aa6f1e9c4a7748'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 7338, 'sha1': '022fc18aa97a30a2f3803bcc4f7d8f3544371a09', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_eae914f097640d21427554e7f6aa6f1e9c4a7748/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': 'eae914f097640d21427554e7f6aa6f1e9c4a7748', 'targets': {'smithi183.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKX/Lm8n4pB52qJdUdfLbrooLRiRZrAIg+jORRiQmF+LaN66a4DgOu9uW2B6A2tbD9+uN1ol8rmfSmTXPcjkVyA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '022fc18aa97a30a2f3803bcc4f7d8f3544371a09', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['rbd/test_lock_fence.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-27_08:05:55', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-27T08:43:54.958 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-27T08:43:55.005 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -d -m0777 -- /var/run/ceph 2024-03-27T08:43:55.075 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-27T08:43:55.075 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/scratch_devs of=/dev/stdout 2024-03-27T08:43:55.130 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-27T08:43:55.130 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_1 2024-03-27T08:43:55.184 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-27T08:43:55.184 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T08:43:55.184 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-27T08:43:55.184 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T08:43:55.184 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T08:43:55.184 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-27 08:43:50.144472415 +0000 2024-03-27T08:43:55.184 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-27 08:40:25.591095300 +0000 2024-03-27T08:43:55.184 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-27 08:40:25.591095300 +0000 2024-03-27T08:43:55.185 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-27 08:40:25.591095300 +0000 2024-03-27T08:43:55.185 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-27T08:43:55.249 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-27T08:43:55.249 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-27T08:43:55.249 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000206328 s, 2.5 MB/s 2024-03-27T08:43:55.250 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-27T08:43:55.306 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_2 2024-03-27T08:43:55.361 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-27T08:43:55.361 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T08:43:55.361 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-27T08:43:55.361 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T08:43:55.361 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T08:43:55.361 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-27 08:43:50.144472415 +0000 2024-03-27T08:43:55.361 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-27 08:40:25.923086275 +0000 2024-03-27T08:43:55.361 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-27 08:40:25.923086275 +0000 2024-03-27T08:43:55.362 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-27 08:40:25.923086275 +0000 2024-03-27T08:43:55.362 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-27T08:43:55.427 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-27T08:43:55.427 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-27T08:43:55.427 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000228304 s, 2.2 MB/s 2024-03-27T08:43:55.428 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-27T08:43:55.484 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_3 2024-03-27T08:43:55.538 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-27T08:43:55.538 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T08:43:55.538 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-27T08:43:55.538 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T08:43:55.538 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T08:43:55.539 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-27 08:43:50.145472389 +0000 2024-03-27T08:43:55.539 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-27 08:40:26.267076924 +0000 2024-03-27T08:43:55.539 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-27 08:40:26.267076924 +0000 2024-03-27T08:43:55.539 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-27 08:40:26.267076924 +0000 2024-03-27T08:43:55.539 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-27T08:43:55.604 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-27T08:43:55.604 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-27T08:43:55.604 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000253368 s, 2.0 MB/s 2024-03-27T08:43:55.605 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-27T08:43:55.661 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_4 2024-03-27T08:43:55.716 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-27T08:43:55.716 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T08:43:55.716 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-27T08:43:55.716 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T08:43:55.716 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T08:43:55.716 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-27 08:43:50.145472389 +0000 2024-03-27T08:43:55.716 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-27 08:40:26.604067763 +0000 2024-03-27T08:43:55.716 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-27 08:40:26.604067763 +0000 2024-03-27T08:43:55.716 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-27 08:40:26.604067763 +0000 2024-03-27T08:43:55.716 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-27T08:43:55.781 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-27T08:43:55.781 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-27T08:43:55.781 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000177135 s, 2.9 MB/s 2024-03-27T08:43:55.782 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-27T08:43:55.838 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-03-27T08:43:55.838 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi183.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-03-27T08:43:55.838 INFO:tasks.ceph:Generating config... 2024-03-27T08:43:55.839 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-27T08:43:55.839 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-27T08:43:55.839 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-27T08:43:55.839 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-27T08:43:55.839 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-27T08:43:55.840 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-27T08:43:55.840 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-27T08:43:55.840 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-27T08:43:55.840 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-03-27T08:43:55.840 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-27T08:43:55.840 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-03-27T08:43:55.840 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-03-27T08:43:55.840 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-27T08:43:55.840 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-27T08:43:55.840 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-27T08:43:55.840 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-03-27T08:43:55.840 INFO:tasks.ceph:[global] ms client mode = crc 2024-03-27T08:43:55.840 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-03-27T08:43:55.840 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-03-27T08:43:55.840 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-03-27T08:43:55.841 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-03-27T08:43:55.841 INFO:tasks.ceph:[global] ms service mode = crc 2024-03-27T08:43:55.841 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-27T08:43:55.841 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-27T08:43:55.841 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-1 2024-03-27T08:43:55.841 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 2-3 2024-03-27T08:43:55.841 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 4-5 2024-03-27T08:43:55.841 INFO:tasks.ceph:Setting up mon.a... 2024-03-27T08:43:55.841 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-27T08:43:55.938 INFO:teuthology.orchestra.run.smithi183.stdout:creating /etc/ceph/ceph.keyring 2024-03-27T08:43:55.941 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-27T08:43:55.993 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-27T08:43:56.019 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.183')] 2024-03-27T08:43:56.020 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.183', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}} 2024-03-27T08:43:56.020 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-27T08:43:56.021 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-27T08:43:56.074 DEBUG:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.183 --print /home/ubuntu/cephtest/ceph.monmap 2024-03-27T08:43:56.164 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:43:56.170+0000 7ff73ae7cf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:43:56.164 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:43:56.170+0000 7ff73ae7cf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:43:56.164 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:43:56.170+0000 7ff73ae7cf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:43:56.164 INFO:teuthology.orchestra.run.smithi183.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-27T08:43:56.165 INFO:teuthology.orchestra.run.smithi183.stdout:monmaptool: generated fsid 1754364f-d979-4d19-889a-ca88fa39bfbf 2024-03-27T08:43:56.165 INFO:teuthology.orchestra.run.smithi183.stdout:setting min_mon_release = quincy 2024-03-27T08:43:56.165 INFO:teuthology.orchestra.run.smithi183.stdout:epoch 0 2024-03-27T08:43:56.165 INFO:teuthology.orchestra.run.smithi183.stdout:fsid 1754364f-d979-4d19-889a-ca88fa39bfbf 2024-03-27T08:43:56.165 INFO:teuthology.orchestra.run.smithi183.stdout:last_changed 2024-03-27T08:43:56.171431+0000 2024-03-27T08:43:56.165 INFO:teuthology.orchestra.run.smithi183.stdout:created 2024-03-27T08:43:56.171431+0000 2024-03-27T08:43:56.165 INFO:teuthology.orchestra.run.smithi183.stdout:min_mon_release 17 (quincy) 2024-03-27T08:43:56.165 INFO:teuthology.orchestra.run.smithi183.stdout:election_strategy: 1 2024-03-27T08:43:56.165 INFO:teuthology.orchestra.run.smithi183.stdout:0: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.a 2024-03-27T08:43:56.165 INFO:teuthology.orchestra.run.smithi183.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-03-27T08:43:56.166 DEBUG:teuthology.orchestra.run.smithi183:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-27T08:43:56.180 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 1754364f-d979-4d19-889a-ca88fa39bfbf... 2024-03-27T08:43:56.181 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-27T08:43:56.271 INFO:teuthology.orchestra.run.smithi183.stdout:[global] 2024-03-27T08:43:56.271 INFO:teuthology.orchestra.run.smithi183.stdout: chdir = "" 2024-03-27T08:43:56.271 INFO:teuthology.orchestra.run.smithi183.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-27T08:43:56.271 INFO:teuthology.orchestra.run.smithi183.stdout: auth supported = cephx 2024-03-27T08:43:56.271 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:56.271 INFO:teuthology.orchestra.run.smithi183.stdout: filestore xattr use omap = true 2024-03-27T08:43:56.271 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:56.271 INFO:teuthology.orchestra.run.smithi183.stdout: mon clock drift allowed = 1.000 2024-03-27T08:43:56.271 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:56.271 INFO:teuthology.orchestra.run.smithi183.stdout: osd crush chooseleaf type = 0 2024-03-27T08:43:56.271 INFO:teuthology.orchestra.run.smithi183.stdout: auth debug = true 2024-03-27T08:43:56.272 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:56.272 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on old message = true 2024-03-27T08:43:56.272 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on bug = true 2024-03-27T08:43:56.272 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:56.272 INFO:teuthology.orchestra.run.smithi183.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-27T08:43:56.272 INFO:teuthology.orchestra.run.smithi183.stdout: mon pg warn max object skew = 0 2024-03-27T08:43:56.272 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:56.272 INFO:teuthology.orchestra.run.smithi183.stdout: # disable pg_autoscaler by default for new pools 2024-03-27T08:43:56.272 INFO:teuthology.orchestra.run.smithi183.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-27T08:43:56.272 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:56.272 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default size = 2 2024-03-27T08:43:56.272 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:56.272 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow primary affinity = true 2024-03-27T08:43:56.272 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow pg remap = true 2024-03-27T08:43:56.272 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on legacy crush tunables = false 2024-03-27T08:43:56.272 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on crush straw calc version zero = false 2024-03-27T08:43:56.272 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on no sortbitwise = false 2024-03-27T08:43:56.273 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on osd down out interval zero = false 2024-03-27T08:43:56.273 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on too few osds = false 2024-03-27T08:43:56.273 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-27T08:43:56.273 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-27T08:43:56.273 INFO:teuthology.orchestra.run.smithi183.stdout: mon_allow_pool_size_one = true 2024-03-27T08:43:56.273 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:56.273 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-27T08:43:56.273 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:56.273 INFO:teuthology.orchestra.run.smithi183.stdout: osd default data pool replay window = 5 2024-03-27T08:43:56.273 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:56.273 INFO:teuthology.orchestra.run.smithi183.stdout: mon allow pool delete = true 2024-03-27T08:43:56.273 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:56.273 INFO:teuthology.orchestra.run.smithi183.stdout: mon cluster log file level = debug 2024-03-27T08:43:56.273 INFO:teuthology.orchestra.run.smithi183.stdout: debug asserts on shutdown = true 2024-03-27T08:43:56.273 INFO:teuthology.orchestra.run.smithi183.stdout: mon health detail to clog = false 2024-03-27T08:43:56.273 INFO:teuthology.orchestra.run.smithi183.stdout: mon host = 172.21.15.183 2024-03-27T08:43:56.273 INFO:teuthology.orchestra.run.smithi183.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-03-27T08:43:56.274 INFO:teuthology.orchestra.run.smithi183.stdout: ms client mode = crc 2024-03-27T08:43:56.274 INFO:teuthology.orchestra.run.smithi183.stdout: ms cluster mode = crc 2024-03-27T08:43:56.274 INFO:teuthology.orchestra.run.smithi183.stdout: ms mon client mode = crc 2024-03-27T08:43:56.274 INFO:teuthology.orchestra.run.smithi183.stdout: ms mon cluster mode = crc 2024-03-27T08:43:56.274 INFO:teuthology.orchestra.run.smithi183.stdout: ms mon service mode = crc 2024-03-27T08:43:56.274 INFO:teuthology.orchestra.run.smithi183.stdout: ms service mode = crc 2024-03-27T08:43:56.274 INFO:teuthology.orchestra.run.smithi183.stdout: fsid = 1754364f-d979-4d19-889a-ca88fa39bfbf 2024-03-27T08:43:56.274 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:56.274 INFO:teuthology.orchestra.run.smithi183.stdout:[osd] 2024-03-27T08:43:56.274 INFO:teuthology.orchestra.run.smithi183.stdout: osd journal size = 100 2024-03-27T08:43:56.274 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:56.274 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub load threshold = 5.0 2024-03-27T08:43:56.274 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub max interval = 600 2024-03-27T08:43:56.274 INFO:teuthology.orchestra.run.smithi183.stdout: osd mclock profile = high_recovery_ops 2024-03-27T08:43:56.274 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:56.274 INFO:teuthology.orchestra.run.smithi183.stdout: osd recover clone overlap = true 2024-03-27T08:43:56.275 INFO:teuthology.orchestra.run.smithi183.stdout: osd recovery max chunk = 1048576 2024-03-27T08:43:56.275 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:56.275 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug shutdown = true 2024-03-27T08:43:56.275 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug op order = true 2024-03-27T08:43:56.275 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug verify stray on activate = true 2024-03-27T08:43:56.275 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug trim objects = true 2024-03-27T08:43:56.275 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:56.275 INFO:teuthology.orchestra.run.smithi183.stdout: osd open classes on start = true 2024-03-27T08:43:56.275 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug pg log writeout = true 2024-03-27T08:43:56.275 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:56.275 INFO:teuthology.orchestra.run.smithi183.stdout: osd deep scrub update digest min age = 30 2024-03-27T08:43:56.275 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:56.275 INFO:teuthology.orchestra.run.smithi183.stdout: osd map max advance = 10 2024-03-27T08:43:56.275 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:56.275 INFO:teuthology.orchestra.run.smithi183.stdout: journal zero on create = true 2024-03-27T08:43:56.275 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:56.275 INFO:teuthology.orchestra.run.smithi183.stdout: filestore ondisk finisher threads = 3 2024-03-27T08:43:56.276 INFO:teuthology.orchestra.run.smithi183.stdout: filestore apply finisher threads = 3 2024-03-27T08:43:56.276 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:56.276 INFO:teuthology.orchestra.run.smithi183.stdout: bdev debug aio = true 2024-03-27T08:43:56.276 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug misdirected ops = true 2024-03-27T08:43:56.276 INFO:teuthology.orchestra.run.smithi183.stdout: debug monc = 20 2024-03-27T08:43:56.276 INFO:teuthology.orchestra.run.smithi183.stdout: crimson alien thread cpu cores = 6-7 2024-03-27T08:43:56.276 INFO:teuthology.orchestra.run.smithi183.stdout: crimson osd obc lru size = 10 2024-03-27T08:43:56.276 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-27T08:43:56.276 INFO:teuthology.orchestra.run.smithi183.stdout: debug osd = 20 2024-03-27T08:43:56.276 INFO:teuthology.orchestra.run.smithi183.stdout: osd shutdown pgref assert = True 2024-03-27T08:43:56.276 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:56.276 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr] 2024-03-27T08:43:56.276 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-27T08:43:56.276 INFO:teuthology.orchestra.run.smithi183.stdout: debug mgr = 20 2024-03-27T08:43:56.276 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2024-03-27T08:43:56.276 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2024-03-27T08:43:56.276 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2024-03-27T08:43:56.277 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2024-03-27T08:43:56.277 INFO:teuthology.orchestra.run.smithi183.stdout: mgr/telemetry/nag = false 2024-03-27T08:43:56.277 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:56.277 INFO:teuthology.orchestra.run.smithi183.stdout:[mon] 2024-03-27T08:43:56.277 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-27T08:43:56.277 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2024-03-27T08:43:56.277 INFO:teuthology.orchestra.run.smithi183.stdout: debug paxos = 20 2024-03-27T08:43:56.277 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2024-03-27T08:43:56.277 INFO:teuthology.orchestra.run.smithi183.stdout: mon data avail warn = 5 2024-03-27T08:43:56.277 INFO:teuthology.orchestra.run.smithi183.stdout: mon mgr mkfs grace = 240 2024-03-27T08:43:56.277 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2024-03-27T08:43:56.277 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd reporter subtree level = osd 2024-03-27T08:43:56.277 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd prime pg temp = true 2024-03-27T08:43:56.277 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2024-03-27T08:43:56.277 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:56.277 INFO:teuthology.orchestra.run.smithi183.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-27T08:43:56.278 INFO:teuthology.orchestra.run.smithi183.stdout: auth mon ticket ttl = 660 # 11m 2024-03-27T08:43:56.278 INFO:teuthology.orchestra.run.smithi183.stdout: auth service ticket ttl = 240 # 4m 2024-03-27T08:43:56.278 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:56.278 INFO:teuthology.orchestra.run.smithi183.stdout: # don't complain about insecure global_id in the test suite 2024-03-27T08:43:56.278 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-27T08:43:56.278 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-27T08:43:56.278 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:56.278 INFO:teuthology.orchestra.run.smithi183.stdout: # 1m isn't quite enough 2024-03-27T08:43:56.278 INFO:teuthology.orchestra.run.smithi183.stdout: mon_down_mkfs_grace = 2m 2024-03-27T08:43:56.278 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:56.278 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_filestore_osds = false 2024-03-27T08:43:56.278 INFO:teuthology.orchestra.run.smithi183.stdout: mon min osdmap epochs = 50 2024-03-27T08:43:56.278 INFO:teuthology.orchestra.run.smithi183.stdout: mon osdmap full prune interval = 2 2024-03-27T08:43:56.278 INFO:teuthology.orchestra.run.smithi183.stdout: mon osdmap full prune min = 15 2024-03-27T08:43:56.278 INFO:teuthology.orchestra.run.smithi183.stdout: mon osdmap full prune txsize = 2 2024-03-27T08:43:56.278 INFO:teuthology.orchestra.run.smithi183.stdout: paxos service trim min = 10 2024-03-27T08:43:56.278 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default crimson = True 2024-03-27T08:43:56.279 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:43:56.279 INFO:teuthology.orchestra.run.smithi183.stdout:[client] 2024-03-27T08:43:56.279 INFO:teuthology.orchestra.run.smithi183.stdout: rgw cache enabled = true 2024-03-27T08:43:56.279 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable ops log = true 2024-03-27T08:43:56.279 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable usage log = true 2024-03-27T08:43:56.279 INFO:teuthology.orchestra.run.smithi183.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-27T08:43:56.279 INFO:teuthology.orchestra.run.smithi183.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-27T08:43:56.279 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.a] 2024-03-27T08:43:56.279 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.0] 2024-03-27T08:43:56.279 INFO:teuthology.orchestra.run.smithi183.stdout: crimson seastar cpu cores = 0-1 2024-03-27T08:43:56.279 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.1] 2024-03-27T08:43:56.279 INFO:teuthology.orchestra.run.smithi183.stdout: crimson seastar cpu cores = 2-3 2024-03-27T08:43:56.279 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.2] 2024-03-27T08:43:56.279 INFO:teuthology.orchestra.run.smithi183.stdout: crimson seastar cpu cores = 4-5 2024-03-27T08:43:56.285 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-27T08:43:56.285 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-27T08:43:56.390 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-27T08:43:56.390 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-27T08:43:56.390 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-27T08:43:56.405 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-27T08:43:56.405 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-27T08:43:56.459 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi183.front.sepia.ceph.com 2024-03-27T08:43:56.459 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-27T08:43:56.459 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-27T08:43:56.459 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-27T08:43:56.538 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-27T08:43:56.538 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-27T08:43:56.592 INFO:tasks.ceph:Setting up mon nodes... 2024-03-27T08:43:56.592 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-27T08:43:56.592 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-27T08:43:56.695 INFO:teuthology.orchestra.run.smithi183.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-27T08:43:56.699 INFO:tasks.ceph:Setting up mds nodes... 2024-03-27T08:43:56.699 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-27T08:43:56.699 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-27T08:43:56.749 INFO:teuthology.orchestra.run.smithi183.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-27T08:43:56.765 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-27T08:43:56.765 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi183.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-03-27T08:43:56.765 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-27T08:43:56.791 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-03-27T08:43:56.792 INFO:tasks.ceph:role: osd.0 2024-03-27T08:43:56.792 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-27T08:43:56.792 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-27T08:43:56.865 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T08:43:56.865 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T08:43:56.865 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T08:43:56.866 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T08:43:56.866 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T08:43:56.866 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-03-27T08:43:56.866 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T08:43:56.866 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T08:43:56.866 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T08:43:56.866 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T08:43:57.374 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-03-27T08:43:57.376 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-03-27T08:43:57.376 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-27T08:43:57.490 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-27T08:43:57.524 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-27T08:43:57.591 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-03-27T08:43:57.591 INFO:tasks.ceph:role: osd.1 2024-03-27T08:43:57.591 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-27T08:43:57.591 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-27T08:43:57.665 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T08:43:57.665 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T08:43:57.665 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T08:43:57.665 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T08:43:57.665 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T08:43:57.665 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-03-27T08:43:57.665 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T08:43:57.665 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T08:43:57.665 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T08:43:57.665 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T08:43:58.131 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-03-27T08:43:58.133 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-03-27T08:43:58.133 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-27T08:43:58.226 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-27T08:43:58.256 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-27T08:43:58.322 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-03-27T08:43:58.322 INFO:tasks.ceph:role: osd.2 2024-03-27T08:43:58.322 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-27T08:43:58.322 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-27T08:43:58.396 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T08:43:58.397 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T08:43:58.397 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T08:43:58.397 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T08:43:58.397 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T08:43:58.397 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-03-27T08:43:58.397 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T08:43:58.397 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T08:43:58.397 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T08:43:58.397 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T08:43:58.881 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-03-27T08:43:58.883 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-03-27T08:43:58.883 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-27T08:43:59.020 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-27T08:43:59.053 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T08:43:59.366 INFO:teuthology.orchestra.run.smithi183.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-27T08:43:59.366 INFO:teuthology.orchestra.run.smithi183.stderr:INFO 2024-03-27 08:43:59,372 seastar - Reactor backend: linux-aio 2024-03-27T08:43:59.376 INFO:teuthology.orchestra.run.smithi183.stderr:==29206==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-27T08:43:59.494 INFO:teuthology.orchestra.run.smithi183.stderr:INFO 2024-03-27 08:43:59,501 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-03-27T08:43:59.495 INFO:teuthology.orchestra.run.smithi183.stderr:INFO 2024-03-27 08:43:59,501 [shard 0:main] osd - get_early_config: set --cpuset 0-1 2024-03-27T08:43:59.495 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG 2024-03-27 08:43:59,502 [shard 0:main] seastar - reactor::drain 2024-03-27T08:44:00.052 INFO:teuthology.orchestra.run.smithi183.stderr:----------------------------------------------------- 2024-03-27T08:44:00.052 INFO:teuthology.orchestra.run.smithi183.stderr:Suppressions used: 2024-03-27T08:44:00.052 INFO:teuthology.orchestra.run.smithi183.stderr: count bytes template 2024-03-27T08:44:00.052 INFO:teuthology.orchestra.run.smithi183.stderr: 1 8 MallocExtension::Register 2024-03-27T08:44:00.052 INFO:teuthology.orchestra.run.smithi183.stderr:----------------------------------------------------- 2024-03-27T08:44:00.052 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-27T08:44:00.110 INFO:teuthology.orchestra.run.smithi183.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-27T08:44:00.114 INFO:teuthology.orchestra.run.smithi183.stderr:INFO 2024-03-27 08:44:00,121 seastar - Reactor backend: linux-aio 2024-03-27T08:44:00.122 INFO:teuthology.orchestra.run.smithi183.stderr:==29204==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-27T08:44:03.563 INFO:teuthology.orchestra.run.smithi183.stderr:----------------------------------------------------- 2024-03-27T08:44:03.563 INFO:teuthology.orchestra.run.smithi183.stderr:Suppressions used: 2024-03-27T08:44:03.563 INFO:teuthology.orchestra.run.smithi183.stderr: count bytes template 2024-03-27T08:44:03.563 INFO:teuthology.orchestra.run.smithi183.stderr: 1 8 MallocExtension::Register 2024-03-27T08:44:03.564 INFO:teuthology.orchestra.run.smithi183.stderr:----------------------------------------------------- 2024-03-27T08:44:03.564 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-27T08:44:03.564 INFO:teuthology.orchestra.run.smithi183.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 1754364f-d979-4d19-889a-ca88fa39bfbf 2024-03-27T08:44:03.590 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-27T08:44:03.617 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T08:44:03.903 INFO:teuthology.orchestra.run.smithi183.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-27T08:44:03.904 INFO:teuthology.orchestra.run.smithi183.stderr:INFO 2024-03-27 08:44:03,911 seastar - Reactor backend: linux-aio 2024-03-27T08:44:03.909 INFO:teuthology.orchestra.run.smithi183.stderr:==30012==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-27T08:44:04.027 INFO:teuthology.orchestra.run.smithi183.stderr:INFO 2024-03-27 08:44:04,034 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-03-27T08:44:04.027 INFO:teuthology.orchestra.run.smithi183.stderr:INFO 2024-03-27 08:44:04,034 [shard 0:main] osd - get_early_config: set --cpuset 2-3 2024-03-27T08:44:04.028 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG 2024-03-27 08:44:04,035 [shard 0:main] seastar - reactor::drain 2024-03-27T08:44:04.563 INFO:teuthology.orchestra.run.smithi183.stderr:----------------------------------------------------- 2024-03-27T08:44:04.563 INFO:teuthology.orchestra.run.smithi183.stderr:Suppressions used: 2024-03-27T08:44:04.563 INFO:teuthology.orchestra.run.smithi183.stderr: count bytes template 2024-03-27T08:44:04.563 INFO:teuthology.orchestra.run.smithi183.stderr: 1 8 MallocExtension::Register 2024-03-27T08:44:04.563 INFO:teuthology.orchestra.run.smithi183.stderr:----------------------------------------------------- 2024-03-27T08:44:04.563 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-27T08:44:04.624 INFO:teuthology.orchestra.run.smithi183.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-27T08:44:04.628 INFO:teuthology.orchestra.run.smithi183.stderr:INFO 2024-03-27 08:44:04,635 seastar - Reactor backend: linux-aio 2024-03-27T08:44:04.635 INFO:teuthology.orchestra.run.smithi183.stderr:==30010==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-27T08:44:04.733 INFO:teuthology.orchestra.run.smithi183.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb99f029 0xb871af0 0xb87306c 0xb873282 0xb8733d8 0xb8738a1 0x54daf 0x5b8477c 0x5db3d29 0x5db3f4d 0x5db4223 0x5e02a63 0x5e02c69 0x5d81750 0x5d81c84 0x5bc6c6f 0x5be1d33 0x6ebc2e4 0x6ecdab7 0x6ecef54 0x6ecefd9 0x6ecf0db 0x6ecf1d3 0x6ecf3e0 0x6ecf5c6 0x6ecf915 0x6ecfcb9 0x6ecfee9 0x6ed0153 0x6ed0420 0x6ed063e 0x6ed06f6 0x6ed07bc 0x6ed089d 0x5a2d276 0x5a2d380 0x5a2d43c 0x5a2d4f8 0x5a2d5b4 0x5a2d701 0x5a2d88c 0x5a2dc4d 0x5a4a6ca 0xb884775 0xb89ea0f 0xb93fa0d 0xb94105b 0xb61d7c3 0xb61e13f 0x368058a 0x3feaf 0x3ff5f 0x346c444 2024-03-27T08:44:04.733 INFO:teuthology.orchestra.run.smithi183.stderr:kernel callstack: 2024-03-27T08:44:08.084 INFO:teuthology.orchestra.run.smithi183.stderr:----------------------------------------------------- 2024-03-27T08:44:08.084 INFO:teuthology.orchestra.run.smithi183.stderr:Suppressions used: 2024-03-27T08:44:08.084 INFO:teuthology.orchestra.run.smithi183.stderr: count bytes template 2024-03-27T08:44:08.084 INFO:teuthology.orchestra.run.smithi183.stderr: 1 8 MallocExtension::Register 2024-03-27T08:44:08.084 INFO:teuthology.orchestra.run.smithi183.stderr:----------------------------------------------------- 2024-03-27T08:44:08.084 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-27T08:44:08.084 INFO:teuthology.orchestra.run.smithi183.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 1754364f-d979-4d19-889a-ca88fa39bfbf 2024-03-27T08:44:08.108 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-27T08:44:08.135 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T08:44:08.425 INFO:teuthology.orchestra.run.smithi183.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-27T08:44:08.426 INFO:teuthology.orchestra.run.smithi183.stderr:INFO 2024-03-27 08:44:08,432 seastar - Reactor backend: linux-aio 2024-03-27T08:44:08.431 INFO:teuthology.orchestra.run.smithi183.stderr:==30818==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-27T08:44:08.548 INFO:teuthology.orchestra.run.smithi183.stderr:INFO 2024-03-27 08:44:08,554 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-03-27T08:44:08.548 INFO:teuthology.orchestra.run.smithi183.stderr:INFO 2024-03-27 08:44:08,555 [shard 0:main] osd - get_early_config: set --cpuset 4-5 2024-03-27T08:44:08.548 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG 2024-03-27 08:44:08,555 [shard 0:main] seastar - reactor::drain 2024-03-27T08:44:09.079 INFO:teuthology.orchestra.run.smithi183.stderr:----------------------------------------------------- 2024-03-27T08:44:09.079 INFO:teuthology.orchestra.run.smithi183.stderr:Suppressions used: 2024-03-27T08:44:09.079 INFO:teuthology.orchestra.run.smithi183.stderr: count bytes template 2024-03-27T08:44:09.079 INFO:teuthology.orchestra.run.smithi183.stderr: 1 8 MallocExtension::Register 2024-03-27T08:44:09.080 INFO:teuthology.orchestra.run.smithi183.stderr:----------------------------------------------------- 2024-03-27T08:44:09.080 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-27T08:44:09.137 INFO:teuthology.orchestra.run.smithi183.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-27T08:44:09.141 INFO:teuthology.orchestra.run.smithi183.stderr:INFO 2024-03-27 08:44:09,148 seastar - Reactor backend: linux-aio 2024-03-27T08:44:09.148 INFO:teuthology.orchestra.run.smithi183.stderr:==30816==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-27T08:44:12.580 INFO:teuthology.orchestra.run.smithi183.stderr:----------------------------------------------------- 2024-03-27T08:44:12.580 INFO:teuthology.orchestra.run.smithi183.stderr:Suppressions used: 2024-03-27T08:44:12.580 INFO:teuthology.orchestra.run.smithi183.stderr: count bytes template 2024-03-27T08:44:12.580 INFO:teuthology.orchestra.run.smithi183.stderr: 1 8 MallocExtension::Register 2024-03-27T08:44:12.580 INFO:teuthology.orchestra.run.smithi183.stderr:----------------------------------------------------- 2024-03-27T08:44:12.580 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-27T08:44:12.581 INFO:teuthology.orchestra.run.smithi183.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 1754364f-d979-4d19-889a-ca88fa39bfbf 2024-03-27T08:44:12.604 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-27T08:44:12.632 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-27T08:44:12.632 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-27T08:44:12.632 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-27T08:44:12.698 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-27T08:44:12.698 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-27T08:44:12.764 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-27T08:44:12.764 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-27T08:44:12.830 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-27T08:44:12.830 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-27T08:44:12.896 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-27T08:44:12.896 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-27T08:44:12.950 INFO:tasks.ceph:Adding keys to all mons... 2024-03-27T08:44:12.950 DEBUG:teuthology.orchestra.run.smithi183:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-27T08:44:13.016 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr.x] 2024-03-27T08:44:13.016 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBM3ANmh2PoKRAAiz4MvhnnauT3GiYUo9qHZg== 2024-03-27T08:44:13.016 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.0] 2024-03-27T08:44:13.016 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBQ3ANmzmAhEBAAFDd3OYYRxHm04hSn9RIMyw== 2024-03-27T08:44:13.016 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.1] 2024-03-27T08:44:13.016 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBU3ANmDuQmLxAATds4lqUts2AbK2b9M/Bepw== 2024-03-27T08:44:13.016 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.2] 2024-03-27T08:44:13.016 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBZ3ANmpKiCERAAjVdRBIHlYIObrU7iCCW5fQ== 2024-03-27T08:44:13.016 INFO:teuthology.orchestra.run.smithi183.stdout:[client.0] 2024-03-27T08:44:13.016 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBM3ANm8gQcLRAA2DbdW3T8WstpF3CmH8zwiA== 2024-03-27T08:44:13.017 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-27T08:44:13.119 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T08:44:13.180 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T08:44:13.242 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T08:44:13.303 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-27T08:44:13.367 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-27T08:44:13.368 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-27T08:44:13.394 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-27T08:44:13.510 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:13.515+0000 7fada01d9a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:13.510 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:13.515+0000 7fada01d9a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:13.524 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-27T08:44:13.550 DEBUG:teuthology.orchestra.run.smithi183:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-27T08:44:13.604 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-27T08:44:13.604 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-27T08:44:13.604 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-27T08:44:13.646 INFO:tasks.ceph.mon.a:Started 2024-03-27T08:44:13.646 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-27T08:44:13.647 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-27T08:44:13.647 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-27T08:44:13.648 INFO:tasks.ceph.mgr.x:Started 2024-03-27T08:44:13.649 INFO:tasks.ceph:Setting crush tunables to default 2024-03-27T08:44:13.649 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd crush tunables default 2024-03-27T08:44:13.731 INFO:tasks.ceph.mon.a.smithi183.stderr:2024-03-27T08:44:13.735+0000 7f3d408baa80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:13.731 INFO:tasks.ceph.mon.a.smithi183.stderr:2024-03-27T08:44:13.735+0000 7f3d408baa80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:13.731 INFO:tasks.ceph.mon.a.smithi183.stderr:2024-03-27T08:44:13.735+0000 7f3d408baa80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:13.731 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:44:13.736+0000 7fdfa0d93000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:13.763 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:13.761+0000 7fc2abd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:13.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:13.794+0000 7fc2abd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:13.859 INFO:teuthology.orchestra.run.smithi183.stderr:adjusted tunables profile to default 2024-03-27T08:44:13.871 INFO:tasks.ceph:check_enable_crimson: True 2024-03-27T08:44:13.871 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-03-27T08:44:13.871 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-03-27T08:44:13.936 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:44:13.941+0000 7fdfa0d93000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:13.936 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:44:13.941+0000 7fdfa0d93000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:13.981 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:13.982+0000 7f01b72ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:13.984 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:13.989+0000 7f01b72ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:14.067 INFO:tasks.ceph.mon.a.smithi183.stderr:2024-03-27T08:44:14.072+0000 7f3d375f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-03-27T08:44:14.067 INFO:tasks.ceph.mon.a.smithi183.stderr:Please be aware that this feature is experimental, untested, 2024-03-27T08:44:14.067 INFO:tasks.ceph.mon.a.smithi183.stderr:unsupported, and may result in data corruption, data loss, 2024-03-27T08:44:14.067 INFO:tasks.ceph.mon.a.smithi183.stderr:and/or irreparable damage to your cluster. Do not use 2024-03-27T08:44:14.067 INFO:tasks.ceph.mon.a.smithi183.stderr:feature with important data. 2024-03-27T08:44:14.067 INFO:tasks.ceph.mon.a.smithi183.stderr: 2024-03-27T08:44:14.414 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:44:14.420+0000 7fdfa0d93000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-27T08:44:14.466 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:44:14.472+0000 7fdfa0d93000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-27T08:44:14.872 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-27T08:44:14.873 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-27T08:44:14.873 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-27T08:44:14.900 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-27T08:44:14.900 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-27T08:44:14.967 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-27T08:44:14.967 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-27T08:44:14.989 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:44:14.995+0000 7fdfa0d93000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-27T08:44:15.015 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 89e61e3d-380d-4d20-9bf5-2ca2b5aeaf1b 0 2024-03-27T08:44:15.046 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:44:15.052+0000 7fdfa0d93000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-27T08:44:15.095 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:44:15.101+0000 7fdfa0d93000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-27T08:44:15.157 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:15.157+0000 7fdcebc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:15.158 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:44:15.164+0000 7fdfa0d93000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-27T08:44:15.160 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:15.166+0000 7fdcebc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:15.202 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:44:15.208+0000 7fdfa0d93000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-27T08:44:15.240 INFO:teuthology.orchestra.run.smithi183.stdout:0 2024-03-27T08:44:15.251 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new f7c37c03-8b44-4e5a-999b-4adbdba14ca4 1 2024-03-27T08:44:15.326 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:44:15.332+0000 7fdfa0d93000 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-27T08:44:15.361 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:15.361+0000 7efec035d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:15.365 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:15.371+0000 7efec035d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:15.400 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:44:15.405+0000 7fdfa0d93000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-27T08:44:15.445 INFO:teuthology.orchestra.run.smithi183.stdout:1 2024-03-27T08:44:15.447 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:44:15.453+0000 7fdfa0d93000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-27T08:44:15.456 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new fe56652e-f3db-482a-a860-e0a10d0e485a 2 2024-03-27T08:44:15.503 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:44:15.509+0000 7fdfa0d93000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-27T08:44:15.554 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:44:15.560+0000 7fdfa0d93000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-27T08:44:15.566 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:15.566+0000 7f7f16e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:15.569 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:15.575+0000 7f7f16e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:15.649 INFO:teuthology.orchestra.run.smithi183.stdout:2 2024-03-27T08:44:15.661 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-03-27T08:44:15.693 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-27T08:44:15.693 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-03-27T08:44:15.735 INFO:tasks.ceph.osd.0:Started 2024-03-27T08:44:15.735 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-03-27T08:44:15.766 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-27T08:44:15.766 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-03-27T08:44:15.808 INFO:tasks.ceph.osd.1:Started 2024-03-27T08:44:15.808 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-03-27T08:44:15.840 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-27T08:44:15.840 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-03-27T08:44:15.882 INFO:tasks.ceph.osd.2:Started 2024-03-27T08:44:15.883 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T08:44:15.980 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:44:15.985+0000 7fdfa0d93000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-27T08:44:16.029 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:16.030+0000 7fb4bea2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:16.033 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:16.039+0000 7fb4bea2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:16.132 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:44:16.137+0000 7fdfa0d93000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-27T08:44:16.147 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:44:16.147 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":6,"fsid":"1754364f-d979-4d19-889a-ca88fa39bfbf","created":"2024-03-27T08:44:13.784474+0000","modified":"2024-03-27T08:44:15.655678+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T08:44:15.655678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"89e61e3d-380d-4d20-9bf5-2ca2b5aeaf1b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f7c37c03-8b44-4e5a-999b-4adbdba14ca4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fe56652e-f3db-482a-a860-e0a10d0e485a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T08:44:16.161 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-27T08:44:16.161 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-27T08:44:16.262 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:44:16.267+0000 7fdfa0d93000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-27T08:44:16.310 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:44:16.316+0000 7fdfa0d93000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-27T08:44:16.431 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:44:16.437+0000 7fdfa0d93000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-27T08:44:16.463 DEBUG:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T08:44:16.539 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:44:16.545+0000 7fdfa0d93000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-27T08:44:16.567 INFO:teuthology.misc.health.smithi183.stderr:2024-03-27T08:44:16.569+0000 7ff55dc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:16.572 INFO:teuthology.misc.health.smithi183.stderr:2024-03-27T08:44:16.577+0000 7ff55dc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:16.640 INFO:teuthology.misc.health.smithi183.stdout: 2024-03-27T08:44:16.640 INFO:teuthology.misc.health.smithi183.stdout:{"epoch":6,"fsid":"1754364f-d979-4d19-889a-ca88fa39bfbf","created":"2024-03-27T08:44:13.784474+0000","modified":"2024-03-27T08:44:15.655678+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T08:44:15.655678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"89e61e3d-380d-4d20-9bf5-2ca2b5aeaf1b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f7c37c03-8b44-4e5a-999b-4adbdba14ca4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fe56652e-f3db-482a-a860-e0a10d0e485a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T08:44:16.652 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-03-27T08:44:16.712 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:44:16.718+0000 7fdfa0d93000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-27T08:44:16.759 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:44:16.765+0000 7fdfa0d93000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-27T08:44:16.930 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:44:16.936+0000 7fdfa0d93000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-27T08:44:17.791 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:44:17.797+0000 7fdfa0d93000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-27T08:44:18.082 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:44:18.088+0000 7fdfa0d93000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-27T08:44:18.141 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:44:18.146+0000 7fdfa0d93000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-27T08:44:18.272 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:44:18.278+0000 7fdfa0d93000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-27T08:44:19.293 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:44:19.299+0000 7fdf3ff30640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-03-27T08:44:22.954 DEBUG:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T08:44:23.065 INFO:teuthology.misc.health.smithi183.stderr:2024-03-27T08:44:23.067+0000 7f734a62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:23.069 INFO:teuthology.misc.health.smithi183.stderr:2024-03-27T08:44:23.075+0000 7f734a62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:23.257 INFO:teuthology.misc.health.smithi183.stdout: 2024-03-27T08:44:23.257 INFO:teuthology.misc.health.smithi183.stdout:{"epoch":9,"fsid":"1754364f-d979-4d19-889a-ca88fa39bfbf","created":"2024-03-27T08:44:13.784474+0000","modified":"2024-03-27T08:44:22.365745+0000","last_up_change":"2024-03-27T08:44:20.359491+0000","last_in_change":"2024-03-27T08:44:15.655678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T08:44:22.327086+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"89e61e3d-380d-4d20-9bf5-2ca2b5aeaf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6805","nonce":2535313537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2535313537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2535313537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6815","nonce":2535313537}]},"public_addr":"172.21.15.183:6805/2535313537","cluster_addr":"172.21.15.183:6806/2535313537","heartbeat_back_addr":"172.21.15.183:6814/2535313537","heartbeat_front_addr":"172.21.15.183:6815/2535313537","state":["exists","up"]},{"osd":1,"uuid":"f7c37c03-8b44-4e5a-999b-4adbdba14ca4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":1280144647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6803","nonce":1280144647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1280144647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6809","nonce":1280144647}]},"public_addr":"172.21.15.183:6802/1280144647","cluster_addr":"172.21.15.183:6803/1280144647","heartbeat_back_addr":"172.21.15.183:6808/1280144647","heartbeat_front_addr":"172.21.15.183:6809/1280144647","state":["exists","up"]},{"osd":2,"uuid":"fe56652e-f3db-482a-a860-e0a10d0e485a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6807","nonce":259299674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":259299674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":259299674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6811","nonce":259299674}]},"public_addr":"172.21.15.183:6807/259299674","cluster_addr":"172.21.15.183:6804/259299674","heartbeat_back_addr":"172.21.15.183:6812/259299674","heartbeat_front_addr":"172.21.15.183:6811/259299674","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T08:44:23.269 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-03-27T08:44:23.269 INFO:tasks.ceph:Creating RBD pool 2024-03-27T08:44:23.269 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd pool create rbd 8 2024-03-27T08:44:23.381 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:23.379+0000 7fe1b5e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:23.385 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:23.391+0000 7fe1b5e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:24.376 INFO:teuthology.orchestra.run.smithi183.stderr:pool 'rbd' created 2024-03-27T08:44:24.399 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-03-27T08:44:24.543 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:24.537+0000 7fbead42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:24.547 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:24.553+0000 7fbead42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:25.379 INFO:teuthology.orchestra.run.smithi183.stderr:enabled application 'rbd' on pool 'rbd' 2024-03-27T08:44:25.399 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-27T08:44:25.400 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-27T08:44:25.400 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-27T08:44:25.400 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-27T08:44:25.400 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-27T08:44:25.547 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:25.549+0000 7f444d59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:25.550 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:25.556+0000 7f444d59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:25.784 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:44:25.796 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":225197696},{"type":"v1","addr":"172.21.15.183:6801","nonce":225197696}]},"active_addr":"172.21.15.183:6801/225197696","active_change":"2024-03-27T08:44:18.291950+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.183:0","nonce":968609885}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.183:0","nonce":395834314}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.183:0","nonce":3559025224}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.183:0","nonce":3329573643}]}]} 2024-03-27T08:44:25.798 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-27T08:44:25.798 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-27T08:44:25.798 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T08:44:25.907 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:25.910+0000 7f12bce24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:25.911 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:25.917+0000 7f12bce24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:26.099 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:44:26.099 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":12,"fsid":"1754364f-d979-4d19-889a-ca88fa39bfbf","created":"2024-03-27T08:44:13.784474+0000","modified":"2024-03-27T08:44:25.382619+0000","last_up_change":"2024-03-27T08:44:20.359491+0000","last_in_change":"2024-03-27T08:44:15.655678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T08:44:22.327086+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-27T08:44:23.597009+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"89e61e3d-380d-4d20-9bf5-2ca2b5aeaf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6805","nonce":2535313537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2535313537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2535313537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6815","nonce":2535313537}]},"public_addr":"172.21.15.183:6805/2535313537","cluster_addr":"172.21.15.183:6806/2535313537","heartbeat_back_addr":"172.21.15.183:6814/2535313537","heartbeat_front_addr":"172.21.15.183:6815/2535313537","state":["exists","up"]},{"osd":1,"uuid":"f7c37c03-8b44-4e5a-999b-4adbdba14ca4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":1280144647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6803","nonce":1280144647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1280144647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6809","nonce":1280144647}]},"public_addr":"172.21.15.183:6802/1280144647","cluster_addr":"172.21.15.183:6803/1280144647","heartbeat_back_addr":"172.21.15.183:6808/1280144647","heartbeat_front_addr":"172.21.15.183:6809/1280144647","state":["exists","up"]},{"osd":2,"uuid":"fe56652e-f3db-482a-a860-e0a10d0e485a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6807","nonce":259299674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":259299674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":259299674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6811","nonce":259299674}]},"public_addr":"172.21.15.183:6807/259299674","cluster_addr":"172.21.15.183:6804/259299674","heartbeat_back_addr":"172.21.15.183:6812/259299674","heartbeat_front_addr":"172.21.15.183:6811/259299674","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T08:44:26.111 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-27T08:44:26.112 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T08:44:26.222 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:26.224+0000 7fe225ff1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:26.225 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:26.231+0000 7fe225ff1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:26.417 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:44:26.418 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":13,"fsid":"1754364f-d979-4d19-889a-ca88fa39bfbf","created":"2024-03-27T08:44:13.784474+0000","modified":"2024-03-27T08:44:26.410657+0000","last_up_change":"2024-03-27T08:44:20.359491+0000","last_in_change":"2024-03-27T08:44:15.655678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T08:44:22.327086+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-27T08:44:23.597009+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"89e61e3d-380d-4d20-9bf5-2ca2b5aeaf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6805","nonce":2535313537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2535313537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2535313537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6815","nonce":2535313537}]},"public_addr":"172.21.15.183:6805/2535313537","cluster_addr":"172.21.15.183:6806/2535313537","heartbeat_back_addr":"172.21.15.183:6814/2535313537","heartbeat_front_addr":"172.21.15.183:6815/2535313537","state":["exists","up"]},{"osd":1,"uuid":"f7c37c03-8b44-4e5a-999b-4adbdba14ca4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":1280144647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6803","nonce":1280144647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1280144647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6809","nonce":1280144647}]},"public_addr":"172.21.15.183:6802/1280144647","cluster_addr":"172.21.15.183:6803/1280144647","heartbeat_back_addr":"172.21.15.183:6808/1280144647","heartbeat_front_addr":"172.21.15.183:6809/1280144647","state":["exists","up"]},{"osd":2,"uuid":"fe56652e-f3db-482a-a860-e0a10d0e485a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6807","nonce":259299674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":259299674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":259299674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6811","nonce":259299674}]},"public_addr":"172.21.15.183:6807/259299674","cluster_addr":"172.21.15.183:6804/259299674","heartbeat_back_addr":"172.21.15.183:6812/259299674","heartbeat_front_addr":"172.21.15.183:6811/259299674","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T08:44:26.431 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-27T08:44:26.432 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-27T08:44:26.432 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-27T08:44:26.548 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:26.549+0000 7f74ddc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:26.548 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:26.549+0000 7fa154b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:26.548 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:26.549+0000 7fce7106d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:26.552 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:26.558+0000 7f74ddc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:26.552 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:26.558+0000 7fa154b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:26.552 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:26.558+0000 7fce7106d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:26.586 INFO:teuthology.orchestra.run.smithi183.stdout:34359738374 2024-03-27T08:44:26.587 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-27T08:44:26.587 INFO:teuthology.orchestra.run.smithi183.stdout:34359738374 2024-03-27T08:44:26.588 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-27T08:44:26.588 INFO:teuthology.orchestra.run.smithi183.stdout:34359738374 2024-03-27T08:44:26.589 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-27T08:44:26.743 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:26.744+0000 7f7aa0a13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:26.744 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:26.745+0000 7fa502be7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:26.744 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:26.745+0000 7f8312b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:26.748 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:26.753+0000 7f7aa0a13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:26.748 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:26.754+0000 7fa502be7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:26.748 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:26.754+0000 7f8312b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:26.944 INFO:teuthology.orchestra.run.smithi183.stdout:34359738373 2024-03-27T08:44:26.947 INFO:teuthology.orchestra.run.smithi183.stdout:34359738373 2024-03-27T08:44:26.956 INFO:teuthology.orchestra.run.smithi183.stdout:34359738373 2024-03-27T08:44:26.958 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738373 for osd.0 2024-03-27T08:44:26.961 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738373 for osd.1 2024-03-27T08:44:26.970 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738373 for osd.2 2024-03-27T08:44:27.959 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-27T08:44:27.962 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-27T08:44:27.970 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-27T08:44:28.088 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:28.088+0000 7f1991c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:28.088 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:28.088+0000 7f658819d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:28.092 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:28.098+0000 7f658819d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:28.093 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:28.098+0000 7f1991c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:28.135 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:28.136+0000 7f04809cf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:28.139 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:28.145+0000 7f04809cf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:28.297 INFO:teuthology.orchestra.run.smithi183.stdout:34359738373 2024-03-27T08:44:28.311 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738373 for osd.1 2024-03-27T08:44:28.320 INFO:teuthology.orchestra.run.smithi183.stdout:34359738373 2024-03-27T08:44:28.333 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738373 for osd.0 2024-03-27T08:44:28.358 INFO:teuthology.orchestra.run.smithi183.stdout:34359738373 2024-03-27T08:44:28.371 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738373 for osd.2 2024-03-27T08:44:29.312 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-27T08:44:29.334 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-27T08:44:29.372 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-27T08:44:29.434 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:29.434+0000 7f2d7d26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:29.438 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:29.444+0000 7f2d7d26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:29.476 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:29.476+0000 7f2b66e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:29.481 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:29.486+0000 7f2b66e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:29.515 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:29.516+0000 7f7ea206d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:29.519 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:29.525+0000 7f7ea206d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:29.636 INFO:teuthology.orchestra.run.smithi183.stdout:34359738374 2024-03-27T08:44:29.650 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.1 2024-03-27T08:44:29.650 DEBUG:teuthology.parallel:result is None 2024-03-27T08:44:29.676 INFO:teuthology.orchestra.run.smithi183.stdout:34359738374 2024-03-27T08:44:29.689 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.0 2024-03-27T08:44:29.689 DEBUG:teuthology.parallel:result is None 2024-03-27T08:44:29.715 INFO:teuthology.orchestra.run.smithi183.stdout:34359738374 2024-03-27T08:44:29.729 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.2 2024-03-27T08:44:29.729 DEBUG:teuthology.parallel:result is None 2024-03-27T08:44:29.729 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-27T08:44:29.729 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T08:44:29.879 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:29.881+0000 7fca1d35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:29.883 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:29.888+0000 7fca1d35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:30.065 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:44:30.065 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-03-27T08:44:30.078 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-03-27T08:44:28.297320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":314572800,"kb_used":82692,"kb_used_data":1460,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490108,"statfs":{"total":322122547200,"available":322037870592,"internally_reserved":0,"allocated":1495040,"data_stored":1251956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.888802"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-27T08:44:26.431783+0000","last_change":"2024-03-27T08:44:25.459078+0000","last_active":"2024-03-27T08:44:26.431783+0000","last_peered":"2024-03-27T08:44:26.431783+0000","last_clean":"2024-03-27T08:44:26.431783+0000","last_became_active":"2024-03-27T08:44:25.453369+0000","last_became_peered":"2024-03-27T08:44:25.453369+0000","last_unstale":"2024-03-27T08:44:26.431783+0000","last_undegraded":"2024-03-27T08:44:26.431783+0000","last_fullsized":"2024-03-27T08:44:26.431783+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-27T08:44:26.439191+0000","last_change":"2024-03-27T08:44:25.483496+0000","last_active":"2024-03-27T08:44:26.439191+0000","last_peered":"2024-03-27T08:44:26.439191+0000","last_clean":"2024-03-27T08:44:26.439191+0000","last_became_active":"2024-03-27T08:44:25.475022+0000","last_became_peered":"2024-03-27T08:44:25.475022+0000","last_unstale":"2024-03-27T08:44:26.439191+0000","last_undegraded":"2024-03-27T08:44:26.439191+0000","last_fullsized":"2024-03-27T08:44:26.439191+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-27T08:44:26.433187+0000","last_change":"2024-03-27T08:44:25.481088+0000","last_active":"2024-03-27T08:44:26.433187+0000","last_peered":"2024-03-27T08:44:26.433187+0000","last_clean":"2024-03-27T08:44:26.433187+0000","last_became_active":"2024-03-27T08:44:25.474410+0000","last_became_peered":"2024-03-27T08:44:25.474410+0000","last_unstale":"2024-03-27T08:44:26.433187+0000","last_undegraded":"2024-03-27T08:44:26.433187+0000","last_fullsized":"2024-03-27T08:44:26.433187+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-27T08:44:26.440767+0000","last_change":"2024-03-27T08:44:25.479583+0000","last_active":"2024-03-27T08:44:26.440767+0000","last_peered":"2024-03-27T08:44:26.440767+0000","last_clean":"2024-03-27T08:44:26.440767+0000","last_became_active":"2024-03-27T08:44:25.470000+0000","last_became_peered":"2024-03-27T08:44:25.470000+0000","last_unstale":"2024-03-27T08:44:26.440767+0000","last_undegraded":"2024-03-27T08:44:26.440767+0000","last_fullsized":"2024-03-27T08:44:26.440767+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T08:44:25.476938+0000","last_change":"2024-03-27T08:44:25.478167+0000","last_active":"2024-03-27T08:44:25.476938+0000","last_peered":"2024-03-27T08:44:25.476938+0000","last_clean":"2024-03-27T08:44:24.373626+0000","last_became_active":"2024-03-27T08:44:25.471316+0000","last_became_peered":"2024-03-27T08:44:25.471316+0000","last_unstale":"2024-03-27T08:44:25.476938+0000","last_undegraded":"2024-03-27T08:44:25.476938+0000","last_fullsized":"2024-03-27T08:44:25.476938+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T08:44:25.484474+0000","last_change":"2024-03-27T08:44:25.485752+0000","last_active":"2024-03-27T08:44:25.484474+0000","last_peered":"2024-03-27T08:44:25.484474+0000","last_clean":"2024-03-27T08:44:24.373626+0000","last_became_active":"2024-03-27T08:44:25.480169+0000","last_became_peered":"2024-03-27T08:44:25.480169+0000","last_unstale":"2024-03-27T08:44:25.484474+0000","last_undegraded":"2024-03-27T08:44:25.484474+0000","last_fullsized":"2024-03-27T08:44:25.484474+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T08:44:25.473853+0000","last_change":"2024-03-27T08:44:25.475056+0000","last_active":"2024-03-27T08:44:25.473853+0000","last_peered":"2024-03-27T08:44:25.473853+0000","last_clean":"2024-03-27T08:44:24.373626+0000","last_became_active":"2024-03-27T08:44:25.470049+0000","last_became_peered":"2024-03-27T08:44:25.470049+0000","last_unstale":"2024-03-27T08:44:25.473853+0000","last_undegraded":"2024-03-27T08:44:25.473853+0000","last_fullsized":"2024-03-27T08:44:25.473853+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-27T08:44:26.439811+0000","last_change":"2024-03-27T08:44:25.454247+0000","last_active":"2024-03-27T08:44:26.439811+0000","last_peered":"2024-03-27T08:44:26.439811+0000","last_clean":"2024-03-27T08:44:26.439811+0000","last_became_active":"2024-03-27T08:44:25.449637+0000","last_became_peered":"2024-03-27T08:44:25.449637+0000","last_unstale":"2024-03-27T08:44:26.439811+0000","last_undegraded":"2024-03-27T08:44:26.439811+0000","last_fullsized":"2024-03-27T08:44:26.439811+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":144,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-27T08:44:26.434334+0000","last_change":"2024-03-27T08:44:23.414190+0000","last_active":"2024-03-27T08:44:26.434334+0000","last_peered":"2024-03-27T08:44:26.434334+0000","last_clean":"2024-03-27T08:44:26.434334+0000","last_became_active":"2024-03-27T08:44:23.409794+0000","last_became_peered":"2024-03-27T08:44:23.409794+0000","last_unstale":"2024-03-27T08:44:26.434334+0000","last_undegraded":"2024-03-27T08:44:26.434334+0000","last_fullsized":"2024-03-27T08:44:26.434334+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:22.365745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:22.365745+0000","last_clean_scrub_stamp":"2024-03-27T08:44:22.365745+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27052,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830548,"statfs":{"total":107374182400,"available":107346481152,"internally_reserved":0,"allocated":102400,"data_stored":23740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27820,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829780,"statfs":{"total":107374182400,"available":107345694720,"internally_reserved":0,"allocated":696320,"data_stored":614108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27820,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829780,"statfs":{"total":107374182400,"available":107345694720,"internally_reserved":0,"allocated":696320,"data_stored":614108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-27T08:44:30.079 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T08:44:30.186 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:30.189+0000 7f2ed602d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:30.190 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:30.196+0000 7f2ed602d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:30.374 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:44:30.375 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-03-27T08:44:30.387 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-03-27T08:44:30.297696+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":314572800,"kb_used":82692,"kb_used_data":1460,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490108,"statfs":{"total":322122547200,"available":322037870592,"internally_reserved":0,"allocated":1495040,"data_stored":1251956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.889178"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-27T08:44:26.431783+0000","last_change":"2024-03-27T08:44:25.459078+0000","last_active":"2024-03-27T08:44:26.431783+0000","last_peered":"2024-03-27T08:44:26.431783+0000","last_clean":"2024-03-27T08:44:26.431783+0000","last_became_active":"2024-03-27T08:44:25.453369+0000","last_became_peered":"2024-03-27T08:44:25.453369+0000","last_unstale":"2024-03-27T08:44:26.431783+0000","last_undegraded":"2024-03-27T08:44:26.431783+0000","last_fullsized":"2024-03-27T08:44:26.431783+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-27T08:44:26.439191+0000","last_change":"2024-03-27T08:44:25.483496+0000","last_active":"2024-03-27T08:44:26.439191+0000","last_peered":"2024-03-27T08:44:26.439191+0000","last_clean":"2024-03-27T08:44:26.439191+0000","last_became_active":"2024-03-27T08:44:25.475022+0000","last_became_peered":"2024-03-27T08:44:25.475022+0000","last_unstale":"2024-03-27T08:44:26.439191+0000","last_undegraded":"2024-03-27T08:44:26.439191+0000","last_fullsized":"2024-03-27T08:44:26.439191+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-27T08:44:26.433187+0000","last_change":"2024-03-27T08:44:25.481088+0000","last_active":"2024-03-27T08:44:26.433187+0000","last_peered":"2024-03-27T08:44:26.433187+0000","last_clean":"2024-03-27T08:44:26.433187+0000","last_became_active":"2024-03-27T08:44:25.474410+0000","last_became_peered":"2024-03-27T08:44:25.474410+0000","last_unstale":"2024-03-27T08:44:26.433187+0000","last_undegraded":"2024-03-27T08:44:26.433187+0000","last_fullsized":"2024-03-27T08:44:26.433187+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-27T08:44:26.440767+0000","last_change":"2024-03-27T08:44:25.479583+0000","last_active":"2024-03-27T08:44:26.440767+0000","last_peered":"2024-03-27T08:44:26.440767+0000","last_clean":"2024-03-27T08:44:26.440767+0000","last_became_active":"2024-03-27T08:44:25.470000+0000","last_became_peered":"2024-03-27T08:44:25.470000+0000","last_unstale":"2024-03-27T08:44:26.440767+0000","last_undegraded":"2024-03-27T08:44:26.440767+0000","last_fullsized":"2024-03-27T08:44:26.440767+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T08:44:25.476938+0000","last_change":"2024-03-27T08:44:25.478167+0000","last_active":"2024-03-27T08:44:25.476938+0000","last_peered":"2024-03-27T08:44:25.476938+0000","last_clean":"2024-03-27T08:44:24.373626+0000","last_became_active":"2024-03-27T08:44:25.471316+0000","last_became_peered":"2024-03-27T08:44:25.471316+0000","last_unstale":"2024-03-27T08:44:25.476938+0000","last_undegraded":"2024-03-27T08:44:25.476938+0000","last_fullsized":"2024-03-27T08:44:25.476938+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T08:44:25.484474+0000","last_change":"2024-03-27T08:44:25.485752+0000","last_active":"2024-03-27T08:44:25.484474+0000","last_peered":"2024-03-27T08:44:25.484474+0000","last_clean":"2024-03-27T08:44:24.373626+0000","last_became_active":"2024-03-27T08:44:25.480169+0000","last_became_peered":"2024-03-27T08:44:25.480169+0000","last_unstale":"2024-03-27T08:44:25.484474+0000","last_undegraded":"2024-03-27T08:44:25.484474+0000","last_fullsized":"2024-03-27T08:44:25.484474+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T08:44:25.473853+0000","last_change":"2024-03-27T08:44:25.475056+0000","last_active":"2024-03-27T08:44:25.473853+0000","last_peered":"2024-03-27T08:44:25.473853+0000","last_clean":"2024-03-27T08:44:24.373626+0000","last_became_active":"2024-03-27T08:44:25.470049+0000","last_became_peered":"2024-03-27T08:44:25.470049+0000","last_unstale":"2024-03-27T08:44:25.473853+0000","last_undegraded":"2024-03-27T08:44:25.473853+0000","last_fullsized":"2024-03-27T08:44:25.473853+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-27T08:44:26.439811+0000","last_change":"2024-03-27T08:44:25.454247+0000","last_active":"2024-03-27T08:44:26.439811+0000","last_peered":"2024-03-27T08:44:26.439811+0000","last_clean":"2024-03-27T08:44:26.439811+0000","last_became_active":"2024-03-27T08:44:25.449637+0000","last_became_peered":"2024-03-27T08:44:25.449637+0000","last_unstale":"2024-03-27T08:44:26.439811+0000","last_undegraded":"2024-03-27T08:44:26.439811+0000","last_fullsized":"2024-03-27T08:44:26.439811+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":144,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-27T08:44:26.434334+0000","last_change":"2024-03-27T08:44:23.414190+0000","last_active":"2024-03-27T08:44:26.434334+0000","last_peered":"2024-03-27T08:44:26.434334+0000","last_clean":"2024-03-27T08:44:26.434334+0000","last_became_active":"2024-03-27T08:44:23.409794+0000","last_became_peered":"2024-03-27T08:44:23.409794+0000","last_unstale":"2024-03-27T08:44:26.434334+0000","last_undegraded":"2024-03-27T08:44:26.434334+0000","last_fullsized":"2024-03-27T08:44:26.434334+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:22.365745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:22.365745+0000","last_clean_scrub_stamp":"2024-03-27T08:44:22.365745+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27052,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830548,"statfs":{"total":107374182400,"available":107346481152,"internally_reserved":0,"allocated":102400,"data_stored":23740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27820,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829780,"statfs":{"total":107374182400,"available":107345694720,"internally_reserved":0,"allocated":696320,"data_stored":614108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27820,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829780,"statfs":{"total":107374182400,"available":107345694720,"internally_reserved":0,"allocated":696320,"data_stored":614108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-27T08:44:30.388 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-27T08:44:30.388 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-27T08:44:30.388 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-27T08:44:30.388 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-27T08:44:30.497 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:30.500+0000 7fe356d14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:30.501 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:44:30.506+0000 7fe356d14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:44:30.724 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:44:30.724 INFO:teuthology.orchestra.run.smithi183.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-27T08:44:30.737 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-27T08:44:30.737 INFO:teuthology.run_tasks:Running task workunit... 2024-03-27T08:44:30.750 INFO:tasks.workunit:Pulling workunits from ref eae914f097640d21427554e7f6aa6f1e9c4a7748 2024-03-27T08:44:30.751 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-27T08:44:30.751 DEBUG:teuthology.orchestra.run.smithi183:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-27T08:44:30.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T08:44:30.766 INFO:teuthology.orchestra.run.smithi183.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-27T08:44:30.766 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-03-27T08:44:30.820 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-03-27T08:44:30.821 DEBUG:teuthology.orchestra.run.smithi183:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-03-27T08:44:30.874 INFO:tasks.workunit:timeout=3h 2024-03-27T08:44:30.875 INFO:tasks.workunit:cleanup=True 2024-03-27T08:44:30.875 DEBUG:teuthology.orchestra.run.smithi183:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout eae914f097640d21427554e7f6aa6f1e9c4a7748 2024-03-27T08:44:30.930 INFO:tasks.workunit.client.0.smithi183.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-27T08:45:19.475 INFO:tasks.workunit.client.0.smithi183.stderr:Note: switching to 'eae914f097640d21427554e7f6aa6f1e9c4a7748'. 2024-03-27T08:45:19.476 INFO:tasks.workunit.client.0.smithi183.stderr: 2024-03-27T08:45:19.476 INFO:tasks.workunit.client.0.smithi183.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-27T08:45:19.476 INFO:tasks.workunit.client.0.smithi183.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-27T08:45:19.476 INFO:tasks.workunit.client.0.smithi183.stderr:state without impacting any branches by switching back to a branch. 2024-03-27T08:45:19.476 INFO:tasks.workunit.client.0.smithi183.stderr: 2024-03-27T08:45:19.476 INFO:tasks.workunit.client.0.smithi183.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-27T08:45:19.476 INFO:tasks.workunit.client.0.smithi183.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-27T08:45:19.476 INFO:tasks.workunit.client.0.smithi183.stderr: 2024-03-27T08:45:19.476 INFO:tasks.workunit.client.0.smithi183.stderr: git switch -c 2024-03-27T08:45:19.476 INFO:tasks.workunit.client.0.smithi183.stderr: 2024-03-27T08:45:19.476 INFO:tasks.workunit.client.0.smithi183.stderr:Or undo this operation with: 2024-03-27T08:45:19.476 INFO:tasks.workunit.client.0.smithi183.stderr: 2024-03-27T08:45:19.477 INFO:tasks.workunit.client.0.smithi183.stderr: git switch - 2024-03-27T08:45:19.477 INFO:tasks.workunit.client.0.smithi183.stderr: 2024-03-27T08:45:19.477 INFO:tasks.workunit.client.0.smithi183.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-27T08:45:19.477 INFO:tasks.workunit.client.0.smithi183.stderr: 2024-03-27T08:45:19.477 INFO:tasks.workunit.client.0.smithi183.stderr:HEAD is now at eae914f0976 Merge pull request #56449 from afreen23/wip-security 2024-03-27T08:45:19.478 DEBUG:teuthology.orchestra.run.smithi183:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-27T08:45:19.536 INFO:tasks.workunit.client.0.smithi183.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-27T08:45:19.538 INFO:tasks.workunit.client.0.smithi183.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-27T08:45:19.538 INFO:tasks.workunit.client.0.smithi183.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-27T08:45:20.223 INFO:tasks.workunit.client.0.smithi183.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-27T08:45:20.286 INFO:tasks.workunit.client.0.smithi183.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-27T08:45:20.326 INFO:tasks.workunit.client.0.smithi183.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-27T08:45:20.327 INFO:tasks.workunit.client.0.smithi183.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-27T08:45:20.327 INFO:tasks.workunit.client.0.smithi183.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-27T08:45:20.367 INFO:tasks.workunit.client.0.smithi183.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-27T08:45:20.370 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-27T08:45:20.370 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-27T08:45:20.425 INFO:tasks.workunit:Running workunits matching rbd/test_lock_fence.sh on client.0... 2024-03-27T08:45:20.426 INFO:tasks.workunit:Running workunit rbd/test_lock_fence.sh... 2024-03-27T08:45:20.426 DEBUG:teuthology.orchestra.run.smithi183:workunit test rbd/test_lock_fence.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=eae914f097640d21427554e7f6aa6f1e9c4a7748 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_lock_fence.sh 2024-03-27T08:45:20.483 INFO:tasks.workunit.client.0.smithi183.stderr:+ IMAGE=rbdrw-image 2024-03-27T08:45:20.483 INFO:tasks.workunit.client.0.smithi183.stderr:+ LOCKID=rbdrw 2024-03-27T08:45:20.483 INFO:tasks.workunit.client.0.smithi183.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_lock_fence.sh 2024-03-27T08:45:20.484 INFO:tasks.workunit.client.0.smithi183.stderr:+ RELPATH=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/librbd 2024-03-27T08:45:20.484 INFO:tasks.workunit.client.0.smithi183.stderr:+ RBDRW=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/librbd/rbdrw.py 2024-03-27T08:45:20.484 INFO:tasks.workunit.client.0.smithi183.stderr:+ rbd create rbdrw-image --size 10 --image-format 2 --image-shared 2024-03-27T08:45:20.564 INFO:tasks.workunit.client.0.smithi183.stderr:2024-03-27T08:45:20.569+0000 7ff59f4415c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:45:20.565 INFO:tasks.workunit.client.0.smithi183.stderr:2024-03-27T08:45:20.569+0000 7ff59f4415c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:45:20.569 INFO:tasks.workunit.client.0.smithi183.stderr:2024-03-27T08:45:20.574+0000 7ff59f4415c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:45:21.891 INFO:tasks.workunit.client.0.smithi183.stderr:+ iochild=34836 2024-03-27T08:45:21.891 INFO:tasks.workunit.client.0.smithi183.stderr:+ LOCKS='[]' 2024-03-27T08:45:21.892 INFO:tasks.workunit.client.0.smithi183.stderr:+ '[' '[]' == '[]' ']' 2024-03-27T08:45:21.892 INFO:tasks.workunit.client.0.smithi183.stderr:+ python3 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/librbd/rbdrw.py rbdrw-image rbdrw 2024-03-27T08:45:21.892 INFO:tasks.workunit.client.0.smithi183.stderr:++ rbd lock list rbdrw-image --format json 2024-03-27T08:45:21.931 INFO:tasks.workunit.client.0.smithi183.stderr:2024-03-27T08:45:21.936+0000 7f2d3ce5b5c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:45:21.931 INFO:tasks.workunit.client.0.smithi183.stderr:2024-03-27T08:45:21.936+0000 7f2d3ce5b5c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:45:21.936 INFO:tasks.workunit.client.0.smithi183.stderr:2024-03-27T08:45:21.941+0000 7f2d3ce5b5c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:45:22.035 INFO:tasks.workunit.client.0.smithi183.stderr:+ LOCKS='[]' 2024-03-27T08:45:22.035 INFO:tasks.workunit.client.0.smithi183.stderr:+ sleep 1 2024-03-27T08:45:23.036 INFO:tasks.workunit.client.0.smithi183.stderr:+ '[' '[]' == '[]' ']' 2024-03-27T08:45:23.036 INFO:tasks.workunit.client.0.smithi183.stderr:++ rbd lock list rbdrw-image --format json 2024-03-27T08:45:23.077 INFO:tasks.workunit.client.0.smithi183.stderr:2024-03-27T08:45:23.082+0000 7fefcd6625c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:45:23.077 INFO:tasks.workunit.client.0.smithi183.stderr:2024-03-27T08:45:23.082+0000 7fefcd6625c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:45:23.082 INFO:tasks.workunit.client.0.smithi183.stderr:2024-03-27T08:45:23.088+0000 7fefcd6625c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:45:23.201 INFO:tasks.workunit.client.0.smithi183.stderr:+ LOCKS='[{"id":"rbdrw","locker":"client.4208","address":"172.21.15.183:0/1310807013"}]' 2024-03-27T08:45:23.202 INFO:tasks.workunit.client.0.smithi183.stderr:+ sleep 1 2024-03-27T08:45:24.202 INFO:tasks.workunit.client.0.smithi183.stderr:+ '[' '[{"id":"rbdrw","locker":"client.4208","address":"172.21.15.183:0/1310807013"}]' == '[]' ']' 2024-03-27T08:45:24.203 INFO:tasks.workunit.client.0.smithi183.stderr:++ rbd lock list rbdrw-image 2024-03-27T08:45:24.203 INFO:tasks.workunit.client.0.smithi183.stderr:++ tail -1 2024-03-27T08:45:24.203 INFO:tasks.workunit.client.0.smithi183.stderr:++ awk '{print $NF;}' 2024-03-27T08:45:24.239 INFO:tasks.workunit.client.0.smithi183.stderr:2024-03-27T08:45:24.243+0000 7f2ebe26a5c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:45:24.239 INFO:tasks.workunit.client.0.smithi183.stderr:2024-03-27T08:45:24.243+0000 7f2ebe26a5c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:45:24.244 INFO:tasks.workunit.client.0.smithi183.stderr:2024-03-27T08:45:24.248+0000 7f2ebe26a5c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:45:24.365 INFO:tasks.workunit.client.0.smithi183.stderr:+ clientaddr=172.21.15.183:0/1310807013 2024-03-27T08:45:24.365 INFO:tasks.workunit.client.0.smithi183.stderr:++ rbd lock list rbdrw-image 2024-03-27T08:45:24.366 INFO:tasks.workunit.client.0.smithi183.stderr:++ tail -1 2024-03-27T08:45:24.366 INFO:tasks.workunit.client.0.smithi183.stderr:++ awk '{print $1;}' 2024-03-27T08:45:24.402 INFO:tasks.workunit.client.0.smithi183.stderr:2024-03-27T08:45:24.406+0000 7fdebb27a5c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:45:24.402 INFO:tasks.workunit.client.0.smithi183.stderr:2024-03-27T08:45:24.406+0000 7fdebb27a5c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:45:24.407 INFO:tasks.workunit.client.0.smithi183.stderr:2024-03-27T08:45:24.411+0000 7fdebb27a5c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:45:24.528 INFO:tasks.workunit.client.0.smithi183.stderr:+ clientid=client.4208 2024-03-27T08:45:24.529 INFO:tasks.workunit.client.0.smithi183.stderr:+ echo 'clientaddr: 172.21.15.183:0/1310807013' 2024-03-27T08:45:24.529 INFO:tasks.workunit.client.0.smithi183.stdout:clientaddr: 172.21.15.183:0/1310807013 2024-03-27T08:45:24.529 INFO:tasks.workunit.client.0.smithi183.stdout:clientid: client.4208 2024-03-27T08:45:24.529 INFO:tasks.workunit.client.0.smithi183.stderr:+ echo 'clientid: client.4208' 2024-03-27T08:45:24.530 INFO:tasks.workunit.client.0.smithi183.stderr:+ ceph osd blocklist add 172.21.15.183:0/1310807013 2024-03-27T08:45:24.627 INFO:tasks.workunit.client.0.smithi183.stderr:2024-03-27T08:45:24.624+0000 7f1ca9b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:45:24.631 INFO:tasks.workunit.client.0.smithi183.stderr:2024-03-27T08:45:24.635+0000 7f1ca9b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:45:26.789 INFO:tasks.workunit.client.0.smithi183.stderr:blocklisting 172.21.15.183:0/1310807013 until 2024-03-27T09:45:25.995028+0000 (3600 sec) 2024-03-27T08:45:26.804 INFO:tasks.workunit.client.0.smithi183.stderr:+ wait 34836 2024-03-27T08:45:26.844 INFO:tasks.workunit.client.0.smithi183.stderr:+ rbdrw_exitcode=108 2024-03-27T08:45:26.844 INFO:tasks.workunit.client.0.smithi183.stderr:+ '[' 108 '!=' 108 ']' 2024-03-27T08:45:26.845 INFO:tasks.workunit.client.0.smithi183.stdout:rbdrw stopped with ESHUTDOWN 2024-03-27T08:45:26.845 INFO:tasks.workunit.client.0.smithi183.stderr:+ echo 'rbdrw stopped with ESHUTDOWN' 2024-03-27T08:45:26.845 INFO:tasks.workunit.client.0.smithi183.stderr:+ set -e 2024-03-27T08:45:26.845 INFO:tasks.workunit.client.0.smithi183.stderr:+ ceph osd blocklist rm 172.21.15.183:0/1310807013 2024-03-27T08:45:26.932 INFO:tasks.workunit.client.0.smithi183.stderr:2024-03-27T08:45:26.931+0000 7fbf74ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:45:26.936 INFO:tasks.workunit.client.0.smithi183.stderr:2024-03-27T08:45:26.940+0000 7fbf74ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:45:27.914 INFO:tasks.workunit.client.0.smithi183.stderr:172.21.15.183:0/1310807013 isn't blocklisted 2024-03-27T08:45:27.925 INFO:tasks.workunit.client.0.smithi183.stderr:+ rbd lock remove rbdrw-image rbdrw client.4208 2024-03-27T08:45:27.962 INFO:tasks.workunit.client.0.smithi183.stderr:2024-03-27T08:45:27.966+0000 7f1c6aa775c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:45:27.962 INFO:tasks.workunit.client.0.smithi183.stderr:2024-03-27T08:45:27.966+0000 7f1c6aa775c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:45:27.966 INFO:tasks.workunit.client.0.smithi183.stderr:2024-03-27T08:45:27.971+0000 7f1c6aa775c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:45:28.870 INFO:tasks.workunit.client.0.smithi183.stderr:+ sleep 30 2024-03-27T08:45:58.871 INFO:tasks.workunit.client.0.smithi183.stderr:+ rbd rm rbdrw-image 2024-03-27T08:45:58.910 INFO:tasks.workunit.client.0.smithi183.stderr:2024-03-27T08:45:58.914+0000 7fa7234635c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:45:58.910 INFO:tasks.workunit.client.0.smithi183.stderr:2024-03-27T08:45:58.914+0000 7fa7234635c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:45:58.915 INFO:tasks.workunit.client.0.smithi183.stderr:2024-03-27T08:45:58.919+0000 7fa7234635c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:45:59.357 INFO:tasks.workunit.client.0.smithi183.stderr: Removing image: 33% complete... Removing image: 66% complete... Removing image: 100% complete...done. 2024-03-27T08:45:59.361 INFO:tasks.workunit.client.0.smithi183.stdout:OK 2024-03-27T08:45:59.362 INFO:tasks.workunit.client.0.smithi183.stderr:+ echo OK 2024-03-27T08:45:59.363 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-27T08:45:59.363 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-27T08:45:59.431 INFO:tasks.workunit:Stopping ['rbd/test_lock_fence.sh'] on client.0... 2024-03-27T08:45:59.431 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-27T08:45:59.762 DEBUG:teuthology.parallel:result is None 2024-03-27T08:45:59.763 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-27T08:45:59.789 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-27T08:45:59.790 DEBUG:teuthology.orchestra.run.smithi183:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-03-27T08:45:59.860 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-03-27T08:45:59.860 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-27T08:45:59.870 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-27T08:45:59.870 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T08:46:00.010 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:46:00.011+0000 7f33c0f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:46:00.013 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:46:00.018+0000 7f33c0f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:46:00.198 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:46:00.198 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-03-27T08:46:00.210 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-03-27T08:45:58.315104+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594499,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":552,"num_read_kb":1147,"num_write":404,"num_write_kb":2939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":367,"ondisk_log_size":367,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":314572800,"kb_used":82508,"kb_used_data":1652,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490292,"statfs":{"total":322122547200,"available":322038059008,"internally_reserved":0,"allocated":1691648,"data_stored":1318382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002889"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:28.823906+0000","last_change":"2024-03-27T08:45:21.831163+0000","last_active":"2024-03-27T08:45:28.823906+0000","last_peered":"2024-03-27T08:45:28.823906+0000","last_clean":"2024-03-27T08:45:28.823906+0000","last_became_active":"2024-03-27T08:44:25.453369+0000","last_became_peered":"2024-03-27T08:44:25.453369+0000","last_unstale":"2024-03-27T08:45:28.823906+0000","last_undegraded":"2024-03-27T08:45:28.823906+0000","last_fullsized":"2024-03-27T08:45:28.823906+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"15'260","reported_seq":547,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:28.830705+0000","last_change":"2024-03-27T08:45:21.842612+0000","last_active":"2024-03-27T08:45:28.830705+0000","last_peered":"2024-03-27T08:45:28.830705+0000","last_clean":"2024-03-27T08:45:28.830705+0000","last_became_active":"2024-03-27T08:44:25.475022+0000","last_became_peered":"2024-03-27T08:44:25.475022+0000","last_unstale":"2024-03-27T08:45:28.830705+0000","last_undegraded":"2024-03-27T08:45:28.830705+0000","last_fullsized":"2024-03-27T08:45:28.830705+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":260,"log_dups_size":0,"ondisk_log_size":260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4096,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":259,"num_read_kb":1036,"num_write":260,"num_write_kb":1040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"15'1","reported_seq":41,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:28.818284+0000","last_change":"2024-03-27T08:45:21.830524+0000","last_active":"2024-03-27T08:45:28.818284+0000","last_peered":"2024-03-27T08:45:28.818284+0000","last_clean":"2024-03-27T08:45:28.818284+0000","last_became_active":"2024-03-27T08:44:25.474410+0000","last_became_peered":"2024-03-27T08:44:25.474410+0000","last_unstale":"2024-03-27T08:45:28.818284+0000","last_undegraded":"2024-03-27T08:45:28.818284+0000","last_fullsized":"2024-03-27T08:45:28.818284+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"15'1","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:28.829970+0000","last_change":"2024-03-27T08:45:21.843385+0000","last_active":"2024-03-27T08:45:28.829970+0000","last_peered":"2024-03-27T08:45:28.829970+0000","last_clean":"2024-03-27T08:45:28.829970+0000","last_became_active":"2024-03-27T08:44:25.470000+0000","last_became_peered":"2024-03-27T08:44:25.470000+0000","last_unstale":"2024-03-27T08:45:28.829970+0000","last_undegraded":"2024-03-27T08:45:28.829970+0000","last_fullsized":"2024-03-27T08:45:28.829970+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:29.471922+0000","last_change":"2024-03-27T08:45:21.799670+0000","last_active":"2024-03-27T08:45:29.471922+0000","last_peered":"2024-03-27T08:45:29.471922+0000","last_clean":"2024-03-27T08:45:29.471922+0000","last_became_active":"2024-03-27T08:44:25.471316+0000","last_became_peered":"2024-03-27T08:44:25.471316+0000","last_unstale":"2024-03-27T08:45:29.471922+0000","last_undegraded":"2024-03-27T08:45:29.471922+0000","last_fullsized":"2024-03-27T08:45:29.471922+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:29.465461+0000","last_change":"2024-03-27T08:45:25.391798+0000","last_active":"2024-03-27T08:45:29.465461+0000","last_peered":"2024-03-27T08:45:29.465461+0000","last_clean":"2024-03-27T08:45:29.465461+0000","last_became_active":"2024-03-27T08:44:25.480169+0000","last_became_peered":"2024-03-27T08:44:25.480169+0000","last_unstale":"2024-03-27T08:45:29.465461+0000","last_undegraded":"2024-03-27T08:45:29.465461+0000","last_fullsized":"2024-03-27T08:45:29.465461+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:29.461865+0000","last_change":"2024-03-27T08:45:25.388815+0000","last_active":"2024-03-27T08:45:29.461865+0000","last_peered":"2024-03-27T08:45:29.461865+0000","last_clean":"2024-03-27T08:45:29.461865+0000","last_became_active":"2024-03-27T08:44:25.470049+0000","last_became_peered":"2024-03-27T08:44:25.470049+0000","last_unstale":"2024-03-27T08:45:29.461865+0000","last_undegraded":"2024-03-27T08:45:29.461865+0000","last_fullsized":"2024-03-27T08:45:29.461865+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"19'6","reported_seq":92,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:28.858842+0000","last_change":"2024-03-27T08:45:21.842267+0000","last_active":"2024-03-27T08:45:28.858842+0000","last_peered":"2024-03-27T08:45:28.858842+0000","last_clean":"2024-03-27T08:45:28.858842+0000","last_became_active":"2024-03-27T08:44:25.449637+0000","last_became_peered":"2024-03-27T08:44:25.449637+0000","last_unstale":"2024-03-27T08:45:28.858842+0000","last_undegraded":"2024-03-27T08:45:28.858842+0000","last_fullsized":"2024-03-27T08:45:28.858842+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":161,"num_read_kb":14,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":160,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:28.822310+0000","last_change":"2024-03-27T08:44:23.414190+0000","last_active":"2024-03-27T08:45:28.822310+0000","last_peered":"2024-03-27T08:45:28.822310+0000","last_clean":"2024-03-27T08:45:28.822310+0000","last_became_active":"2024-03-27T08:44:23.409794+0000","last_became_peered":"2024-03-27T08:44:23.409794+0000","last_unstale":"2024-03-27T08:45:28.822310+0000","last_undegraded":"2024-03-27T08:45:28.822310+0000","last_fullsized":"2024-03-27T08:45:28.822310+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:22.365745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:22.365745+0000","last_clean_scrub_stamp":"2024-03-27T08:44:22.365745+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4131,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":449,"num_read_kb":1056,"num_write":267,"num_write_kb":1045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":270,"ondisk_log_size":270,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738462,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27108,"kb_used_data":156,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830492,"statfs":{"total":107374182400,"available":107346423808,"internally_reserved":0,"allocated":159744,"data_stored":43128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738462,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27700,"kb_used_data":748,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829900,"statfs":{"total":107374182400,"available":107345817600,"internally_reserved":0,"allocated":765952,"data_stored":637627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27700,"kb_used_data":748,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829900,"statfs":{"total":107374182400,"available":107345817600,"internally_reserved":0,"allocated":765952,"data_stored":637627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-27T08:46:00.211 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T08:46:00.320 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:46:00.321+0000 7f431ef9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:46:00.323 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:46:00.328+0000 7f431ef9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:46:00.509 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:46:00.509 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-03-27T08:46:00.522 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":64,"stamp":"2024-03-27T08:46:00.315482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594499,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":552,"num_read_kb":1147,"num_write":404,"num_write_kb":2939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":367,"ondisk_log_size":367,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":314572800,"kb_used":82508,"kb_used_data":1652,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490292,"statfs":{"total":322122547200,"available":322038059008,"internally_reserved":0,"allocated":1691648,"data_stored":1318382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002801"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:28.823906+0000","last_change":"2024-03-27T08:45:21.831163+0000","last_active":"2024-03-27T08:45:28.823906+0000","last_peered":"2024-03-27T08:45:28.823906+0000","last_clean":"2024-03-27T08:45:28.823906+0000","last_became_active":"2024-03-27T08:44:25.453369+0000","last_became_peered":"2024-03-27T08:44:25.453369+0000","last_unstale":"2024-03-27T08:45:28.823906+0000","last_undegraded":"2024-03-27T08:45:28.823906+0000","last_fullsized":"2024-03-27T08:45:28.823906+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"15'260","reported_seq":547,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:28.830705+0000","last_change":"2024-03-27T08:45:21.842612+0000","last_active":"2024-03-27T08:45:28.830705+0000","last_peered":"2024-03-27T08:45:28.830705+0000","last_clean":"2024-03-27T08:45:28.830705+0000","last_became_active":"2024-03-27T08:44:25.475022+0000","last_became_peered":"2024-03-27T08:44:25.475022+0000","last_unstale":"2024-03-27T08:45:28.830705+0000","last_undegraded":"2024-03-27T08:45:28.830705+0000","last_fullsized":"2024-03-27T08:45:28.830705+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":260,"log_dups_size":0,"ondisk_log_size":260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4096,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":259,"num_read_kb":1036,"num_write":260,"num_write_kb":1040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"15'1","reported_seq":41,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:28.818284+0000","last_change":"2024-03-27T08:45:21.830524+0000","last_active":"2024-03-27T08:45:28.818284+0000","last_peered":"2024-03-27T08:45:28.818284+0000","last_clean":"2024-03-27T08:45:28.818284+0000","last_became_active":"2024-03-27T08:44:25.474410+0000","last_became_peered":"2024-03-27T08:44:25.474410+0000","last_unstale":"2024-03-27T08:45:28.818284+0000","last_undegraded":"2024-03-27T08:45:28.818284+0000","last_fullsized":"2024-03-27T08:45:28.818284+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"15'1","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:28.829970+0000","last_change":"2024-03-27T08:45:21.843385+0000","last_active":"2024-03-27T08:45:28.829970+0000","last_peered":"2024-03-27T08:45:28.829970+0000","last_clean":"2024-03-27T08:45:28.829970+0000","last_became_active":"2024-03-27T08:44:25.470000+0000","last_became_peered":"2024-03-27T08:44:25.470000+0000","last_unstale":"2024-03-27T08:45:28.829970+0000","last_undegraded":"2024-03-27T08:45:28.829970+0000","last_fullsized":"2024-03-27T08:45:28.829970+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:29.471922+0000","last_change":"2024-03-27T08:45:21.799670+0000","last_active":"2024-03-27T08:45:29.471922+0000","last_peered":"2024-03-27T08:45:29.471922+0000","last_clean":"2024-03-27T08:45:29.471922+0000","last_became_active":"2024-03-27T08:44:25.471316+0000","last_became_peered":"2024-03-27T08:44:25.471316+0000","last_unstale":"2024-03-27T08:45:29.471922+0000","last_undegraded":"2024-03-27T08:45:29.471922+0000","last_fullsized":"2024-03-27T08:45:29.471922+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:29.465461+0000","last_change":"2024-03-27T08:45:25.391798+0000","last_active":"2024-03-27T08:45:29.465461+0000","last_peered":"2024-03-27T08:45:29.465461+0000","last_clean":"2024-03-27T08:45:29.465461+0000","last_became_active":"2024-03-27T08:44:25.480169+0000","last_became_peered":"2024-03-27T08:44:25.480169+0000","last_unstale":"2024-03-27T08:45:29.465461+0000","last_undegraded":"2024-03-27T08:45:29.465461+0000","last_fullsized":"2024-03-27T08:45:29.465461+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:29.461865+0000","last_change":"2024-03-27T08:45:25.388815+0000","last_active":"2024-03-27T08:45:29.461865+0000","last_peered":"2024-03-27T08:45:29.461865+0000","last_clean":"2024-03-27T08:45:29.461865+0000","last_became_active":"2024-03-27T08:44:25.470049+0000","last_became_peered":"2024-03-27T08:44:25.470049+0000","last_unstale":"2024-03-27T08:45:29.461865+0000","last_undegraded":"2024-03-27T08:45:29.461865+0000","last_fullsized":"2024-03-27T08:45:29.461865+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"19'6","reported_seq":92,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:28.858842+0000","last_change":"2024-03-27T08:45:21.842267+0000","last_active":"2024-03-27T08:45:28.858842+0000","last_peered":"2024-03-27T08:45:28.858842+0000","last_clean":"2024-03-27T08:45:28.858842+0000","last_became_active":"2024-03-27T08:44:25.449637+0000","last_became_peered":"2024-03-27T08:44:25.449637+0000","last_unstale":"2024-03-27T08:45:28.858842+0000","last_undegraded":"2024-03-27T08:45:28.858842+0000","last_fullsized":"2024-03-27T08:45:28.858842+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":161,"num_read_kb":14,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":160,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:28.822310+0000","last_change":"2024-03-27T08:44:23.414190+0000","last_active":"2024-03-27T08:45:28.822310+0000","last_peered":"2024-03-27T08:45:28.822310+0000","last_clean":"2024-03-27T08:45:28.822310+0000","last_became_active":"2024-03-27T08:44:23.409794+0000","last_became_peered":"2024-03-27T08:44:23.409794+0000","last_unstale":"2024-03-27T08:45:28.822310+0000","last_undegraded":"2024-03-27T08:45:28.822310+0000","last_fullsized":"2024-03-27T08:45:28.822310+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:22.365745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:22.365745+0000","last_clean_scrub_stamp":"2024-03-27T08:44:22.365745+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4131,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":449,"num_read_kb":1056,"num_write":267,"num_write_kb":1045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":270,"ondisk_log_size":270,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738462,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27108,"kb_used_data":156,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830492,"statfs":{"total":107374182400,"available":107346423808,"internally_reserved":0,"allocated":159744,"data_stored":43128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738462,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27700,"kb_used_data":748,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829900,"statfs":{"total":107374182400,"available":107345817600,"internally_reserved":0,"allocated":765952,"data_stored":637627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27700,"kb_used_data":748,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829900,"statfs":{"total":107374182400,"available":107345817600,"internally_reserved":0,"allocated":765952,"data_stored":637627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-27T08:46:00.523 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-27T08:46:00.523 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T08:46:00.634 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:46:00.634+0000 7f3a9991d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:46:00.637 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:46:00.642+0000 7f3a9991d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:46:00.819 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:46:00.820 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-03-27T08:46:00.832 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":64,"stamp":"2024-03-27T08:46:00.315482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594499,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":552,"num_read_kb":1147,"num_write":404,"num_write_kb":2939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":367,"ondisk_log_size":367,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":314572800,"kb_used":82508,"kb_used_data":1652,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490292,"statfs":{"total":322122547200,"available":322038059008,"internally_reserved":0,"allocated":1691648,"data_stored":1318382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002801"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:28.823906+0000","last_change":"2024-03-27T08:45:21.831163+0000","last_active":"2024-03-27T08:45:28.823906+0000","last_peered":"2024-03-27T08:45:28.823906+0000","last_clean":"2024-03-27T08:45:28.823906+0000","last_became_active":"2024-03-27T08:44:25.453369+0000","last_became_peered":"2024-03-27T08:44:25.453369+0000","last_unstale":"2024-03-27T08:45:28.823906+0000","last_undegraded":"2024-03-27T08:45:28.823906+0000","last_fullsized":"2024-03-27T08:45:28.823906+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"15'260","reported_seq":547,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:28.830705+0000","last_change":"2024-03-27T08:45:21.842612+0000","last_active":"2024-03-27T08:45:28.830705+0000","last_peered":"2024-03-27T08:45:28.830705+0000","last_clean":"2024-03-27T08:45:28.830705+0000","last_became_active":"2024-03-27T08:44:25.475022+0000","last_became_peered":"2024-03-27T08:44:25.475022+0000","last_unstale":"2024-03-27T08:45:28.830705+0000","last_undegraded":"2024-03-27T08:45:28.830705+0000","last_fullsized":"2024-03-27T08:45:28.830705+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":260,"log_dups_size":0,"ondisk_log_size":260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4096,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":259,"num_read_kb":1036,"num_write":260,"num_write_kb":1040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"15'1","reported_seq":41,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:28.818284+0000","last_change":"2024-03-27T08:45:21.830524+0000","last_active":"2024-03-27T08:45:28.818284+0000","last_peered":"2024-03-27T08:45:28.818284+0000","last_clean":"2024-03-27T08:45:28.818284+0000","last_became_active":"2024-03-27T08:44:25.474410+0000","last_became_peered":"2024-03-27T08:44:25.474410+0000","last_unstale":"2024-03-27T08:45:28.818284+0000","last_undegraded":"2024-03-27T08:45:28.818284+0000","last_fullsized":"2024-03-27T08:45:28.818284+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"15'1","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:28.829970+0000","last_change":"2024-03-27T08:45:21.843385+0000","last_active":"2024-03-27T08:45:28.829970+0000","last_peered":"2024-03-27T08:45:28.829970+0000","last_clean":"2024-03-27T08:45:28.829970+0000","last_became_active":"2024-03-27T08:44:25.470000+0000","last_became_peered":"2024-03-27T08:44:25.470000+0000","last_unstale":"2024-03-27T08:45:28.829970+0000","last_undegraded":"2024-03-27T08:45:28.829970+0000","last_fullsized":"2024-03-27T08:45:28.829970+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:29.471922+0000","last_change":"2024-03-27T08:45:21.799670+0000","last_active":"2024-03-27T08:45:29.471922+0000","last_peered":"2024-03-27T08:45:29.471922+0000","last_clean":"2024-03-27T08:45:29.471922+0000","last_became_active":"2024-03-27T08:44:25.471316+0000","last_became_peered":"2024-03-27T08:44:25.471316+0000","last_unstale":"2024-03-27T08:45:29.471922+0000","last_undegraded":"2024-03-27T08:45:29.471922+0000","last_fullsized":"2024-03-27T08:45:29.471922+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:29.465461+0000","last_change":"2024-03-27T08:45:25.391798+0000","last_active":"2024-03-27T08:45:29.465461+0000","last_peered":"2024-03-27T08:45:29.465461+0000","last_clean":"2024-03-27T08:45:29.465461+0000","last_became_active":"2024-03-27T08:44:25.480169+0000","last_became_peered":"2024-03-27T08:44:25.480169+0000","last_unstale":"2024-03-27T08:45:29.465461+0000","last_undegraded":"2024-03-27T08:45:29.465461+0000","last_fullsized":"2024-03-27T08:45:29.465461+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:29.461865+0000","last_change":"2024-03-27T08:45:25.388815+0000","last_active":"2024-03-27T08:45:29.461865+0000","last_peered":"2024-03-27T08:45:29.461865+0000","last_clean":"2024-03-27T08:45:29.461865+0000","last_became_active":"2024-03-27T08:44:25.470049+0000","last_became_peered":"2024-03-27T08:44:25.470049+0000","last_unstale":"2024-03-27T08:45:29.461865+0000","last_undegraded":"2024-03-27T08:45:29.461865+0000","last_fullsized":"2024-03-27T08:45:29.461865+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"19'6","reported_seq":92,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:28.858842+0000","last_change":"2024-03-27T08:45:21.842267+0000","last_active":"2024-03-27T08:45:28.858842+0000","last_peered":"2024-03-27T08:45:28.858842+0000","last_clean":"2024-03-27T08:45:28.858842+0000","last_became_active":"2024-03-27T08:44:25.449637+0000","last_became_peered":"2024-03-27T08:44:25.449637+0000","last_unstale":"2024-03-27T08:45:28.858842+0000","last_undegraded":"2024-03-27T08:45:28.858842+0000","last_fullsized":"2024-03-27T08:45:28.858842+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":161,"num_read_kb":14,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":160,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:28.822310+0000","last_change":"2024-03-27T08:44:23.414190+0000","last_active":"2024-03-27T08:45:28.822310+0000","last_peered":"2024-03-27T08:45:28.822310+0000","last_clean":"2024-03-27T08:45:28.822310+0000","last_became_active":"2024-03-27T08:44:23.409794+0000","last_became_peered":"2024-03-27T08:44:23.409794+0000","last_unstale":"2024-03-27T08:45:28.822310+0000","last_undegraded":"2024-03-27T08:45:28.822310+0000","last_fullsized":"2024-03-27T08:45:28.822310+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:22.365745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:22.365745+0000","last_clean_scrub_stamp":"2024-03-27T08:44:22.365745+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4131,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":449,"num_read_kb":1056,"num_write":267,"num_write_kb":1045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":270,"ondisk_log_size":270,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738462,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27108,"kb_used_data":156,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830492,"statfs":{"total":107374182400,"available":107346423808,"internally_reserved":0,"allocated":159744,"data_stored":43128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738462,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27700,"kb_used_data":748,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829900,"statfs":{"total":107374182400,"available":107345817600,"internally_reserved":0,"allocated":765952,"data_stored":637627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27700,"kb_used_data":748,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829900,"statfs":{"total":107374182400,"available":107345817600,"internally_reserved":0,"allocated":765952,"data_stored":637627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-27T08:46:00.833 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T08:46:00.943 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:46:00.943+0000 7fc7529f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:46:00.946 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:46:00.950+0000 7fc7529f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:46:01.133 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:46:01.133 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":19,"fsid":"1754364f-d979-4d19-889a-ca88fa39bfbf","created":"2024-03-27T08:44:13.784474+0000","modified":"2024-03-27T08:45:28.804196+0000","last_up_change":"2024-03-27T08:44:20.359491+0000","last_in_change":"2024-03-27T08:44:15.655678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T08:44:22.327086+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-27T08:44:23.597009+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"89e61e3d-380d-4d20-9bf5-2ca2b5aeaf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6805","nonce":2535313537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2535313537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2535313537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6815","nonce":2535313537}]},"public_addr":"172.21.15.183:6805/2535313537","cluster_addr":"172.21.15.183:6806/2535313537","heartbeat_back_addr":"172.21.15.183:6814/2535313537","heartbeat_front_addr":"172.21.15.183:6815/2535313537","state":["exists","up"]},{"osd":1,"uuid":"f7c37c03-8b44-4e5a-999b-4adbdba14ca4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":1280144647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6803","nonce":1280144647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1280144647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6809","nonce":1280144647}]},"public_addr":"172.21.15.183:6802/1280144647","cluster_addr":"172.21.15.183:6803/1280144647","heartbeat_back_addr":"172.21.15.183:6808/1280144647","heartbeat_front_addr":"172.21.15.183:6809/1280144647","state":["exists","up"]},{"osd":2,"uuid":"fe56652e-f3db-482a-a860-e0a10d0e485a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6807","nonce":259299674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":259299674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":259299674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6811","nonce":259299674}]},"public_addr":"172.21.15.183:6807/259299674","cluster_addr":"172.21.15.183:6804/259299674","heartbeat_back_addr":"172.21.15.183:6812/259299674","heartbeat_front_addr":"172.21.15.183:6811/259299674","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.183:0/1310807013":"2024-03-27T09:45:28.093689+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T08:46:02.148 INFO:tasks.ceph:Scrubbing osd.0 2024-03-27T08:46:02.148 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-27T08:46:02.276 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:46:02.271+0000 7f5691c60640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:46:02.279 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:46:02.283+0000 7f5691c60640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:46:02.292 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-03-27T08:46:02.293 INFO:teuthology.orchestra.run.smithi183.stdout: "success": "" 2024-03-27T08:46:02.293 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-03-27T08:46:02.304 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-27T08:46:02.414 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:46:02.413+0000 7f806de2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:46:02.418 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:46:02.421+0000 7f806de2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:46:02.603 INFO:teuthology.orchestra.run.smithi183.stderr:instructed osd(s) 0 to deep-scrub 2024-03-27T08:46:02.621 INFO:tasks.ceph:Scrubbing osd.1 2024-03-27T08:46:02.622 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-27T08:46:02.735 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:46:02.731+0000 7f4615d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:46:02.738 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:46:02.742+0000 7f4615d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:46:02.753 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-03-27T08:46:02.753 INFO:teuthology.orchestra.run.smithi183.stdout: "success": "" 2024-03-27T08:46:02.753 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-03-27T08:46:02.765 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-27T08:46:02.876 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:46:02.874+0000 7f9d4595d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:46:02.879 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:46:02.883+0000 7f9d4595d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:46:03.064 INFO:teuthology.orchestra.run.smithi183.stderr:instructed osd(s) 1 to deep-scrub 2024-03-27T08:46:03.086 INFO:tasks.ceph:Scrubbing osd.2 2024-03-27T08:46:03.087 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-27T08:46:03.203 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:46:03.198+0000 7f8664e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:46:03.206 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:46:03.210+0000 7f8664e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:46:03.221 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-03-27T08:46:03.222 INFO:teuthology.orchestra.run.smithi183.stdout: "success": "" 2024-03-27T08:46:03.222 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-03-27T08:46:03.233 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-27T08:46:03.342 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:46:03.341+0000 7f81a222d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:46:03.346 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:46:03.350+0000 7f81a222d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:46:03.531 INFO:teuthology.orchestra.run.smithi183.stderr:instructed osd(s) 2 to deep-scrub 2024-03-27T08:46:03.549 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T08:46:03.658 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:46:03.658+0000 7f6135e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:46:03.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:46:03.665+0000 7f6135e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:46:03.843 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:46:03.843 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-03-27T08:46:03.856 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2024-03-27T08:46:02.315848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":641,"num_read_kb":1153,"num_write":410,"num_write_kb":2942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":380,"ondisk_log_size":380,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":314572800,"kb_used":82492,"kb_used_data":1636,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490308,"statfs":{"total":322122547200,"available":322038075392,"internally_reserved":0,"allocated":1675264,"data_stored":1310158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4112,"num_objects":-2,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":89,"num_read_kb":6,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002724"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:28.823906+0000","last_change":"2024-03-27T08:45:21.831163+0000","last_active":"2024-03-27T08:45:28.823906+0000","last_peered":"2024-03-27T08:45:28.823906+0000","last_clean":"2024-03-27T08:45:28.823906+0000","last_became_active":"2024-03-27T08:44:25.453369+0000","last_became_peered":"2024-03-27T08:44:25.453369+0000","last_unstale":"2024-03-27T08:45:28.823906+0000","last_undegraded":"2024-03-27T08:45:28.823906+0000","last_fullsized":"2024-03-27T08:45:28.823906+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"15'260","reported_seq":548,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:28.831149+0000","last_change":"2024-03-27T08:45:21.842612+0000","last_active":"2024-03-27T08:45:28.831149+0000","last_peered":"2024-03-27T08:45:28.831149+0000","last_clean":"2024-03-27T08:45:28.831149+0000","last_became_active":"2024-03-27T08:44:25.475022+0000","last_became_peered":"2024-03-27T08:44:25.475022+0000","last_unstale":"2024-03-27T08:45:28.831149+0000","last_undegraded":"2024-03-27T08:45:28.831149+0000","last_fullsized":"2024-03-27T08:45:28.831149+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":260,"log_dups_size":0,"ondisk_log_size":260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":259,"num_read_kb":1036,"num_write":261,"num_write_kb":1040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"15'1","reported_seq":42,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:28.818788+0000","last_change":"2024-03-27T08:45:21.830524+0000","last_active":"2024-03-27T08:45:28.818788+0000","last_peered":"2024-03-27T08:45:28.818788+0000","last_clean":"2024-03-27T08:45:28.818788+0000","last_became_active":"2024-03-27T08:44:25.474410+0000","last_became_peered":"2024-03-27T08:44:25.474410+0000","last_unstale":"2024-03-27T08:45:28.818788+0000","last_undegraded":"2024-03-27T08:45:28.818788+0000","last_fullsized":"2024-03-27T08:45:28.818788+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":6,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"15'1","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:59.047297+0000","last_change":"2024-03-27T08:45:21.843385+0000","last_active":"2024-03-27T08:45:59.047297+0000","last_peered":"2024-03-27T08:45:59.047297+0000","last_clean":"2024-03-27T08:45:59.047297+0000","last_became_active":"2024-03-27T08:44:25.470000+0000","last_became_peered":"2024-03-27T08:44:25.470000+0000","last_unstale":"2024-03-27T08:45:59.047297+0000","last_undegraded":"2024-03-27T08:45:59.047297+0000","last_fullsized":"2024-03-27T08:45:59.047297+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":43,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:59.102336+0000","last_change":"2024-03-27T08:45:21.799670+0000","last_active":"2024-03-27T08:45:59.102336+0000","last_peered":"2024-03-27T08:45:59.102336+0000","last_clean":"2024-03-27T08:45:59.102336+0000","last_became_active":"2024-03-27T08:44:25.471316+0000","last_became_peered":"2024-03-27T08:44:25.471316+0000","last_unstale":"2024-03-27T08:45:59.102336+0000","last_undegraded":"2024-03-27T08:45:59.102336+0000","last_fullsized":"2024-03-27T08:45:59.102336+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:29.465461+0000","last_change":"2024-03-27T08:45:25.391798+0000","last_active":"2024-03-27T08:45:29.465461+0000","last_peered":"2024-03-27T08:45:29.465461+0000","last_clean":"2024-03-27T08:45:29.465461+0000","last_became_active":"2024-03-27T08:44:25.480169+0000","last_became_peered":"2024-03-27T08:44:25.480169+0000","last_unstale":"2024-03-27T08:45:29.465461+0000","last_undegraded":"2024-03-27T08:45:29.465461+0000","last_fullsized":"2024-03-27T08:45:29.465461+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:29.461865+0000","last_change":"2024-03-27T08:45:25.388815+0000","last_active":"2024-03-27T08:45:29.461865+0000","last_peered":"2024-03-27T08:45:29.461865+0000","last_clean":"2024-03-27T08:45:29.461865+0000","last_became_active":"2024-03-27T08:44:25.470049+0000","last_became_peered":"2024-03-27T08:44:25.470049+0000","last_unstale":"2024-03-27T08:45:29.461865+0000","last_undegraded":"2024-03-27T08:45:29.461865+0000","last_fullsized":"2024-03-27T08:45:29.461865+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"19'19","reported_seq":135,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:59.351560+0000","last_change":"2024-03-27T08:45:21.842267+0000","last_active":"2024-03-27T08:45:59.351560+0000","last_peered":"2024-03-27T08:45:59.351560+0000","last_clean":"2024-03-27T08:45:59.351560+0000","last_became_active":"2024-03-27T08:44:25.449637+0000","last_became_peered":"2024-03-27T08:44:25.449637+0000","last_unstale":"2024-03-27T08:45:59.351560+0000","last_undegraded":"2024-03-27T08:45:59.351560+0000","last_fullsized":"2024-03-27T08:45:59.351560+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:24.373626+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":241,"num_read_kb":20,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":160,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-27T08:45:28.822310+0000","last_change":"2024-03-27T08:44:23.414190+0000","last_active":"2024-03-27T08:45:28.822310+0000","last_peered":"2024-03-27T08:45:28.822310+0000","last_clean":"2024-03-27T08:45:28.822310+0000","last_became_active":"2024-03-27T08:44:23.409794+0000","last_became_peered":"2024-03-27T08:44:23.409794+0000","last_unstale":"2024-03-27T08:45:28.822310+0000","last_undegraded":"2024-03-27T08:45:28.822310+0000","last_fullsized":"2024-03-27T08:45:28.822310+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:44:22.365745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:44:22.365745+0000","last_clean_scrub_stamp":"2024-03-27T08:44:22.365745+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":538,"num_read_kb":1062,"num_write":273,"num_write_kb":1048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":283,"ondisk_log_size":283,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738467,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27108,"kb_used_data":156,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830492,"statfs":{"total":107374182400,"available":107346423808,"internally_reserved":0,"allocated":159744,"data_stored":43128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738467,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27692,"kb_used_data":740,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":757760,"data_stored":633515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27692,"kb_used_data":740,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":757760,"data_stored":633515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-27T08:46:03.860 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-03-27T08:44:24.373626+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=8, tm_min=44, tm_sec=24, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=8, tm_min=46, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T08:46:03.860 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-03-27T08:44:24.373626+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=8, tm_min=44, tm_sec=24, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=8, tm_min=46, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T08:46:03.860 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-03-27T08:44:24.373626+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=8, tm_min=44, tm_sec=24, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=8, tm_min=46, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T08:46:03.860 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-03-27T08:44:24.373626+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=8, tm_min=44, tm_sec=24, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=8, tm_min=46, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T08:46:03.860 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-03-27T08:44:24.373626+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=8, tm_min=44, tm_sec=24, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=8, tm_min=46, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T08:46:03.861 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-03-27T08:44:24.373626+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=8, tm_min=44, tm_sec=24, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=8, tm_min=46, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T08:46:03.861 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-03-27T08:44:24.373626+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=8, tm_min=44, tm_sec=24, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=8, tm_min=46, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T08:46:03.861 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-03-27T08:44:24.373626+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=8, tm_min=44, tm_sec=24, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=8, tm_min=46, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T08:46:03.861 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-27T08:44:22.365745+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=8, tm_min=44, tm_sec=22, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=8, tm_min=46, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T08:46:03.861 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-27T08:46:06.838 INFO:tasks.ceph.mon.a.smithi183.stderr:2024-03-27T08:46:06.843+0000 7f3d39df9640 -1 log_channel(cluster) log [ERR] : Health check failed: 9 scrub errors (OSD_SCRUB_ERRORS) 2024-03-27T08:46:06.839 INFO:tasks.ceph.mon.a.smithi183.stderr:2024-03-27T08:46:06.843+0000 7f3d39df9640 -1 log_channel(cluster) log [ERR] : Health check failed: Possible data damage: 9 pgs inconsistent (PG_DAMAGED) 2024-03-27T08:46:18.319 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:46:18.323+0000 7fdf3a725640 -1 calc_pg_upmaps abort due to max <= 0 2024-03-27T08:46:18.319 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-27T08:46:18.323+0000 7fdf3a725640 -1 calc_pg_upmaps abort due to max <= 0 2024-03-27T08:46:23.863 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T08:46:23.985 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:46:23.986+0000 7f1f93d1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:46:23.989 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:46:23.994+0000 7f1f93d1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:46:24.172 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-27T08:46:24.172 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-03-27T08:46:24.185 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2024-03-27T08:46:22.321219+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":641,"num_read_kb":1153,"num_write":410,"num_write_kb":2942,"num_scrub_errors":9,"num_shallow_scrub_errors":11,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":387,"ondisk_log_size":387,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":314572800,"kb_used":82492,"kb_used_data":1636,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490308,"statfs":{"total":322122547200,"available":322038075392,"internally_reserved":0,"allocated":1675264,"data_stored":1310158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003694"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean+inconsistent","last_fresh":"2024-03-27T08:46:03.075239+0000","last_change":"2024-03-27T08:46:03.094484+0000","last_active":"2024-03-27T08:46:03.075239+0000","last_peered":"2024-03-27T08:46:03.075239+0000","last_clean":"2024-03-27T08:46:03.075239+0000","last_became_active":"2024-03-27T08:44:25.453369+0000","last_became_peered":"2024-03-27T08:44:25.453369+0000","last_unstale":"2024-03-27T08:46:03.075239+0000","last_undegraded":"2024-03-27T08:46:03.075239+0000","last_fullsized":"2024-03-27T08:46:03.075239+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:46:03.094154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:46:03.094154+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":1,"num_shallow_scrub_errors":1,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"19'262","reported_seq":550,"reported_epoch":19,"state":"active+clean+inconsistent","last_fresh":"2024-03-27T08:46:03.078465+0000","last_change":"2024-03-27T08:46:03.098025+0000","last_active":"2024-03-27T08:46:03.078465+0000","last_peered":"2024-03-27T08:46:03.078465+0000","last_clean":"2024-03-27T08:46:03.078465+0000","last_became_active":"2024-03-27T08:44:25.475022+0000","last_became_peered":"2024-03-27T08:44:25.475022+0000","last_unstale":"2024-03-27T08:46:03.078465+0000","last_undegraded":"2024-03-27T08:46:03.078465+0000","last_fullsized":"2024-03-27T08:46:03.078465+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'262","last_scrub_stamp":"2024-03-27T08:46:03.097722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:46:03.097722+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":262,"log_dups_size":0,"ondisk_log_size":262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":259,"num_read_kb":1036,"num_write":261,"num_write_kb":1040,"num_scrub_errors":1,"num_shallow_scrub_errors":1,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"19'3","reported_seq":44,"reported_epoch":19,"state":"active+clean+inconsistent","last_fresh":"2024-03-27T08:46:03.074426+0000","last_change":"2024-03-27T08:46:03.092128+0000","last_active":"2024-03-27T08:46:03.074426+0000","last_peered":"2024-03-27T08:46:03.074426+0000","last_clean":"2024-03-27T08:46:03.074426+0000","last_became_active":"2024-03-27T08:44:25.474410+0000","last_became_peered":"2024-03-27T08:44:25.474410+0000","last_unstale":"2024-03-27T08:46:03.074426+0000","last_undegraded":"2024-03-27T08:46:03.074426+0000","last_fullsized":"2024-03-27T08:46:03.074426+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'3","last_scrub_stamp":"2024-03-27T08:46:03.091910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:46:03.091910+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":6,"num_write":2,"num_write_kb":1,"num_scrub_errors":1,"num_shallow_scrub_errors":1,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"19'2","reported_seq":36,"reported_epoch":19,"state":"active+clean+inconsistent","last_fresh":"2024-03-27T08:46:03.075085+0000","last_change":"2024-03-27T08:46:03.101030+0000","last_active":"2024-03-27T08:46:03.075085+0000","last_peered":"2024-03-27T08:46:03.075085+0000","last_clean":"2024-03-27T08:46:03.075085+0000","last_became_active":"2024-03-27T08:44:25.470000+0000","last_became_peered":"2024-03-27T08:44:25.470000+0000","last_unstale":"2024-03-27T08:46:03.075085+0000","last_undegraded":"2024-03-27T08:46:03.075085+0000","last_fullsized":"2024-03-27T08:46:03.075085+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2024-03-27T08:46:03.100786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:46:03.100786+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":1,"num_shallow_scrub_errors":2,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'3","reported_seq":45,"reported_epoch":19,"state":"active+clean+inconsistent","last_fresh":"2024-03-27T08:46:02.611102+0000","last_change":"2024-03-27T08:46:02.622093+0000","last_active":"2024-03-27T08:46:02.611102+0000","last_peered":"2024-03-27T08:46:02.611102+0000","last_clean":"2024-03-27T08:46:02.611102+0000","last_became_active":"2024-03-27T08:44:25.471316+0000","last_became_peered":"2024-03-27T08:44:25.471316+0000","last_unstale":"2024-03-27T08:46:02.611102+0000","last_undegraded":"2024-03-27T08:46:02.611102+0000","last_fullsized":"2024-03-27T08:46:02.611102+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'3","last_scrub_stamp":"2024-03-27T08:46:02.621688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:46:02.621688+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":1,"num_shallow_scrub_errors":1,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"19'1","reported_seq":28,"reported_epoch":19,"state":"active+clean+inconsistent","last_fresh":"2024-03-27T08:46:03.540192+0000","last_change":"2024-03-27T08:46:03.548338+0000","last_active":"2024-03-27T08:46:03.540192+0000","last_peered":"2024-03-27T08:46:03.540192+0000","last_clean":"2024-03-27T08:46:03.540192+0000","last_became_active":"2024-03-27T08:44:25.480169+0000","last_became_peered":"2024-03-27T08:44:25.480169+0000","last_unstale":"2024-03-27T08:46:03.540192+0000","last_undegraded":"2024-03-27T08:46:03.540192+0000","last_fullsized":"2024-03-27T08:46:03.540192+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'1","last_scrub_stamp":"2024-03-27T08:46:03.548108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:46:03.548108+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":1,"num_shallow_scrub_errors":1,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean+inconsistent","last_fresh":"2024-03-27T08:46:03.538652+0000","last_change":"2024-03-27T08:46:03.548740+0000","last_active":"2024-03-27T08:46:03.538652+0000","last_peered":"2024-03-27T08:46:03.538652+0000","last_clean":"2024-03-27T08:46:03.538652+0000","last_became_active":"2024-03-27T08:44:25.470049+0000","last_became_peered":"2024-03-27T08:44:25.470049+0000","last_unstale":"2024-03-27T08:46:03.538652+0000","last_undegraded":"2024-03-27T08:46:03.538652+0000","last_fullsized":"2024-03-27T08:46:03.538652+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T08:46:03.548524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:46:03.548524+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":1,"num_shallow_scrub_errors":1,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"19'19","reported_seq":137,"reported_epoch":19,"state":"active+clean+inconsistent","last_fresh":"2024-03-27T08:46:03.074233+0000","last_change":"2024-03-27T08:46:03.091617+0000","last_active":"2024-03-27T08:46:03.074233+0000","last_peered":"2024-03-27T08:46:03.074233+0000","last_clean":"2024-03-27T08:46:03.074233+0000","last_became_active":"2024-03-27T08:44:25.449637+0000","last_became_peered":"2024-03-27T08:44:25.449637+0000","last_unstale":"2024-03-27T08:46:03.074233+0000","last_undegraded":"2024-03-27T08:46:03.074233+0000","last_fullsized":"2024-03-27T08:46:03.074233+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'19","last_scrub_stamp":"2024-03-27T08:46:03.091362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:46:03.091362+0000","last_clean_scrub_stamp":"2024-03-27T08:44:24.373626+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":241,"num_read_kb":20,"num_write":7,"num_write_kb":4,"num_scrub_errors":1,"num_shallow_scrub_errors":2,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":162,"reported_epoch":19,"state":"active+clean+inconsistent","last_fresh":"2024-03-27T08:46:03.072935+0000","last_change":"2024-03-27T08:46:03.102004+0000","last_active":"2024-03-27T08:46:03.072935+0000","last_peered":"2024-03-27T08:46:03.072935+0000","last_clean":"2024-03-27T08:46:03.072935+0000","last_became_active":"2024-03-27T08:44:23.409794+0000","last_became_peered":"2024-03-27T08:44:23.409794+0000","last_unstale":"2024-03-27T08:46:03.072935+0000","last_undegraded":"2024-03-27T08:46:03.072935+0000","last_fullsized":"2024-03-27T08:46:03.072935+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"11'97","last_scrub_stamp":"2024-03-27T08:46:03.101751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T08:46:03.101751+0000","last_clean_scrub_stamp":"2024-03-27T08:44:22.365745+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":1,"num_shallow_scrub_errors":1,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":538,"num_read_kb":1062,"num_write":273,"num_write_kb":1048,"num_scrub_errors":8,"num_shallow_scrub_errors":10,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":290,"ondisk_log_size":290,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":1,"num_shallow_scrub_errors":1,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738487,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27108,"kb_used_data":156,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830492,"statfs":{"total":107374182400,"available":107346423808,"internally_reserved":0,"allocated":159744,"data_stored":43128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738487,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27692,"kb_used_data":740,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":757760,"data_stored":633515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738487,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27692,"kb_used_data":740,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":757760,"data_stored":633515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-27T08:46:24.186 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-27T08:46:24.296 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:46:24.294+0000 7f6309a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:46:24.300 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-27T08:46:24.304+0000 7f6309a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T08:46:24.506 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-27T08:46:24.506 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-27T08:46:24.506 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-27T08:46:24.506 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T08:46:24.571 INFO:tasks.ceph.osd.0:Stopped 2024-03-27T08:46:24.572 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-27T08:46:24.572 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T08:46:24.638 INFO:tasks.ceph.osd.1:Stopped 2024-03-27T08:46:24.639 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-27T08:46:24.639 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T08:46:24.696 INFO:tasks.ceph.osd.2:Stopped 2024-03-27T08:46:24.696 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-27T08:46:24.697 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-27T08:46:24.697 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T08:46:24.723 INFO:tasks.ceph.mgr.x:Stopped 2024-03-27T08:46:24.724 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-27T08:46:24.724 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-27T08:46:24.724 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T08:46:24.774 INFO:tasks.ceph.mon.a:Stopped 2024-03-27T08:46:24.775 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-27T08:46:24.775 DEBUG:teuthology.orchestra.run.smithi183:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-27T08:46:24.843 INFO:teuthology.orchestra.run.smithi183.stdout:2024-03-27T08:46:06.843999+0000 mon.a (mon.0) 143 : cluster [ERR] Health check failed: 9 scrub errors (OSD_SCRUB_ERRORS) 2024-03-27T08:46:24.843 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-03-27T08:46:24.843 DEBUG:teuthology.orchestra.run.smithi183:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-27T08:46:24.911 DEBUG:teuthology.orchestra.run.smithi183:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-27T08:46:24.979 INFO:teuthology.orchestra.run.smithi183.stdout:2024-03-27T08:46:06.843999+0000 mon.a (mon.0) 143 : cluster [ERR] Health check failed: 9 scrub errors (OSD_SCRUB_ERRORS) 2024-03-27T08:46:24.979 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-27T08:46:24.979 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-27T08:46:25.195 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-27T08:46:25.195 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-27T08:46:25.251 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-27T08:46:25.251 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-27T08:46:25.308 INFO:tasks.ceph:Archiving mon data... 2024-03-27T08:46:25.309 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2024-03-27_08:05:55-crimson-rados-wip-matanb-crimson-foreign-ptr-distro-crimson-smithi/7625904/data/mon.a.tgz 2024-03-27T08:46:25.310 DEBUG:teuthology.orchestra.run.smithi183:> mktemp 2024-03-27T08:46:25.324 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.okh3kXZBlj 2024-03-27T08:46:25.324 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.okh3kXZBlj 2024-03-27T08:46:25.432 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.okh3kXZBlj 2024-03-27T08:46:25.516 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.okh3kXZBlj is 160KB 2024-03-27T08:46:25.568 DEBUG:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.okh3kXZBlj 2024-03-27T08:46:25.583 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-27T08:46:25.583 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-27T08:46:25.703 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-27T08:46:25.703 INFO:tasks.ceph:Archiving crash dumps... 2024-03-27T08:46:25.705 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-03-27_08:05:55-crimson-rados-wip-matanb-crimson-foreign-ptr-distro-crimson-smithi/7625904/remote/smithi183/crash 2024-03-27T08:46:25.706 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-27T08:46:25.769 INFO:tasks.ceph:Compressing logs... 2024-03-27T08:46:25.769 DEBUG:teuthology.orchestra.run.smithi183:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-27T08:46:25.858 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33715.log 2024-03-27T08:46:25.858 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-27T08:46:25.859 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.33715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35543.log 2024-03-27T08:46:25.859 INFO:teuthology.orchestra.run.smithi183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33715.log.gz 2024-03-27T08:46:25.859 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32161.log 2024-03-27T08:46:25.859 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.35543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34304.log 2024-03-27T08:46:25.859 INFO:teuthology.orchestra.run.smithi183.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35543.log.gz 2024-03-27T08:46:25.859 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.32161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34284.log 2024-03-27T08:46:25.859 INFO:teuthology.orchestra.run.smithi183.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32161.log.gz 2024-03-27T08:46:25.859 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34178.log 2024-03-27T08:46:25.859 INFO:teuthology.orchestra.run.smithi183.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34304.log.gz 2024-03-27T08:46:25.859 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34410.log 2024-03-27T08:46:25.859 INFO:teuthology.orchestra.run.smithi183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34284.log.gz 2024-03-27T08:46:25.859 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34283.log 2024-03-27T08:46:25.859 INFO:teuthology.orchestra.run.smithi183.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34178.log.gz 2024-03-27T08:46:25.859 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32429.log 2024-03-27T08:46:25.860 INFO:teuthology.orchestra.run.smithi183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34410.log.gz 2024-03-27T08:46:25.860 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34899.log 2024-03-27T08:46:25.860 INFO:teuthology.orchestra.run.smithi183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34283.log.gz 2024-03-27T08:46:25.860 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.32429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34922.log 2024-03-27T08:46:25.860 INFO:teuthology.orchestra.run.smithi183.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32429.log.gz 2024-03-27T08:46:25.860 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35249.log 2024-03-27T08:46:25.860 INFO:teuthology.orchestra.run.smithi183.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.34899.log.gz 2024-03-27T08:46:25.860 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34052.log 2024-03-27T08:46:25.860 INFO:teuthology.orchestra.run.smithi183.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.34922.log.gz 2024-03-27T08:46:25.860 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.35249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33886.log 2024-03-27T08:46:25.860 INFO:teuthology.orchestra.run.smithi183.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35249.log.gz 2024-03-27T08:46:25.860 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35459.log 2024-03-27T08:46:25.860 INFO:teuthology.orchestra.run.smithi183.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34052.log.gz 2024-03-27T08:46:25.860 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.33886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33771.log 2024-03-27T08:46:25.861 INFO:teuthology.orchestra.run.smithi183.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33886.log.gz 2024-03-27T08:46:25.861 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.35459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35417.log 2024-03-27T08:46:25.861 INFO:teuthology.orchestra.run.smithi183.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35459.log.gz 2024-03-27T08:46:25.861 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.33771.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28732.log 2024-03-27T08:46:25.861 INFO:teuthology.orchestra.run.smithi183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33771.log.gz 2024-03-27T08:46:25.861 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.35417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35013.log 2024-03-27T08:46:25.861 INFO:teuthology.orchestra.run.smithi183.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35417.log.gz 2024-03-27T08:46:25.861 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph.tmp-client.admin.28732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32051.log 2024-03-27T08:46:25.861 INFO:teuthology.orchestra.run.smithi183.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28732.log.gz 2024-03-27T08:46:25.861 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.35013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35165.log 2024-03-27T08:46:25.861 INFO:teuthology.orchestra.run.smithi183.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.35013.log.gz 2024-03-27T08:46:25.861 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.32051.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-27T08:46:25.861 INFO:teuthology.orchestra.run.smithi183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32051.log.gz 2024-03-27T08:46:25.861 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.35165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34051.log 2024-03-27T08:46:25.861 INFO:teuthology.orchestra.run.smithi183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35165.log.gz 2024-03-27T08:46:25.862 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35123.log 2024-03-27T08:46:25.862 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34051.log: 86.6%gzip -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-27T08:46:25.862 INFO:teuthology.orchestra.run.smithi183.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.31963.log 2024-03-27T08:46:25.862 INFO:teuthology.orchestra.run.smithi183.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34051.log.gz 2024-03-27T08:46:25.862 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.35123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34819.log 2024-03-27T08:46:25.862 INFO:teuthology.orchestra.run.smithi183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35123.log.gz 2024-03-27T08:46:25.862 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.31963.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-27T08:46:25.862 INFO:teuthology.orchestra.run.smithi183.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31963.log.gz 2024-03-27T08:46:25.862 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33843.log 2024-03-27T08:46:25.862 INFO:teuthology.orchestra.run.smithi183.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.34819.log.gz 2024-03-27T08:46:25.862 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34992.log 2024-03-27T08:46:25.862 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.33843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34600.log 2024-03-27T08:46:25.862 INFO:teuthology.orchestra.run.smithi183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33843.log.gz 2024-03-27T08:46:25.862 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34177.log 2024-03-27T08:46:25.863 INFO:teuthology.orchestra.run.smithi183.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.34992.log.gz 2024-03-27T08:46:25.863 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34600.log: gzip -5 --verbose 50.0% -- -- replaced with /var/log/ceph/ceph-client.admin.34600.log.gz /var/log/ceph/ceph-client.admin.35501.log 2024-03-27T08:46:25.863 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-27T08:46:25.863 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35291.log 2024-03-27T08:46:25.863 INFO:teuthology.orchestra.run.smithi183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34177.log.gz 2024-03-27T08:46:25.863 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.35501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35333.log 2024-03-27T08:46:25.863 INFO:teuthology.orchestra.run.smithi183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35501.log.gz 2024-03-27T08:46:25.863 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.35291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35207.log 2024-03-27T08:46:25.863 INFO:teuthology.orchestra.run.smithi183.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35291.log.gz 2024-03-27T08:46:25.863 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.35333.log: gzip 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35333.log.gz -5 2024-03-27T08:46:25.863 INFO:teuthology.orchestra.run.smithi183.stderr: --verbose -- /var/log/ceph/ceph-client.admin.32399.log 2024-03-27T08:46:25.863 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.35207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34176.log 2024-03-27T08:46:25.863 INFO:teuthology.orchestra.run.smithi183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35207.log.gz 2024-03-27T08:46:25.863 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.32399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34944.log 2024-03-27T08:46:25.863 INFO:teuthology.orchestra.run.smithi183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32399.log.gz 2024-03-27T08:46:25.863 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34050.log 2024-03-27T08:46:25.864 INFO:teuthology.orchestra.run.smithi183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34176.log.gz 2024-03-27T08:46:25.864 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34836.log 2024-03-27T08:46:25.864 INFO:teuthology.orchestra.run.smithi183.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34944.log.gz 2024-03-27T08:46:25.864 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34390.log 2024-03-27T08:46:25.864 INFO:teuthology.orchestra.run.smithi183.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34050.log.gz 2024-03-27T08:46:25.864 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34836.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-27T08:46:25.864 INFO:teuthology.orchestra.run.smithi183.stderr: 58.4% -- replaced with /var/log/ceph/ceph-client.admin.34836.log.gz 2024-03-27T08:46:25.864 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34968.log 2024-03-27T08:46:25.864 INFO:teuthology.orchestra.run.smithi183.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34390.log.gz 2024-03-27T08:46:25.864 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35585.log 2024-03-27T08:46:25.864 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34877.log 2024-03-27T08:46:25.864 INFO:teuthology.orchestra.run.smithi183.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34968.log.gz 2024-03-27T08:46:25.864 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.35585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32201.log 2024-03-27T08:46:25.864 INFO:teuthology.orchestra.run.smithi183.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35585.log.gz 2024-03-27T08:46:25.864 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34877.log: gzip 66.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.34877.log.gz --verbose 2024-03-27T08:46:25.865 INFO:teuthology.orchestra.run.smithi183.stderr: -- /var/log/ceph/ceph-client.admin.35625.log 2024-03-27T08:46:25.865 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34837.log 2024-03-27T08:46:25.865 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.32201.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32201.log.gz 2024-03-27T08:46:25.865 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.35625.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35625.log.gz 2024-03-27T08:46:25.865 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-27T08:46:25.865 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-27T08:46:25.865 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34837.log: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.34837.log.gz 2024-03-27T08:46:25.865 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-27T08:46:25.865 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33928.log 2024-03-27T08:46:25.865 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.2.log: 85.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34558.log 2024-03-27T08:46:25.865 INFO:teuthology.orchestra.run.smithi183.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-03-27T08:46:25.865 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.33928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35375.log 2024-03-27T08:46:25.865 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34558.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33928.log.gz 2024-03-27T08:46:25.865 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34516.log 2024-03-27T08:46:25.866 INFO:teuthology.orchestra.run.smithi183.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34558.log.gz 2024-03-27T08:46:25.866 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.35375.log: gzip 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35375.log.gz 2024-03-27T08:46:25.866 INFO:teuthology.orchestra.run.smithi183.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.34431.log 2024-03-27T08:46:25.866 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34516.log: 50.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33970.log 2024-03-27T08:46:25.866 INFO:teuthology.orchestra.run.smithi183.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34516.log.gz 2024-03-27T08:46:25.866 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32241.log 2024-03-27T08:46:25.866 INFO:teuthology.orchestra.run.smithi183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34431.log.gz 2024-03-27T08:46:25.866 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.33970.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33970.log.gz 2024-03-27T08:46:25.866 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.32241.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32241.log.gz 2024-03-27T08:46:25.867 INFO:teuthology.orchestra.run.smithi183.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-27T08:46:25.867 INFO:teuthology.orchestra.run.smithi183.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-27T08:46:25.882 INFO:teuthology.orchestra.run.smithi183.stderr: 94.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-27T08:46:25.884 INFO:teuthology.orchestra.run.smithi183.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-27T08:46:25.971 INFO:teuthology.orchestra.run.smithi183.stderr: 95.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-27T08:46:25.973 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-27T08:46:25.973 INFO:teuthology.orchestra.run.smithi183.stderr:real 0m0.149s 2024-03-27T08:46:25.973 INFO:teuthology.orchestra.run.smithi183.stderr:user 0m0.262s 2024-03-27T08:46:25.974 INFO:teuthology.orchestra.run.smithi183.stderr:sys 0m0.045s 2024-03-27T08:46:25.974 INFO:tasks.ceph:Archiving logs... 2024-03-27T08:46:25.974 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/log/ceph to /home/teuthworker/archive/matan-2024-03-27_08:05:55-crimson-rados-wip-matanb-crimson-foreign-ptr-distro-crimson-smithi/7625904/remote/smithi183/log 2024-03-27T08:46:25.975 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-27T08:46:26.173 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-27T08:46:26.187 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-27T08:46:26.188 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-27T08:46:26.239 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-27T08:46:26.250 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-27T08:46:26.250 DEBUG:teuthology.orchestra.run.smithi183:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-27T08:46:26.268 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-03-27T08:46:26.271 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-27T08:46:26.271 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-03-27T08:46:26.272 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 7 +166us[ +166us] +/- 71ms 2024-03-27T08:46:26.272 INFO:teuthology.orchestra.run.smithi183.stdout:^* hv02.front.sepia.ceph.com 3 6 377 7 -382us[ -391us] +/- 80ms 2024-03-27T08:46:26.272 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 8 +200us[ +190us] +/- 83ms 2024-03-27T08:46:26.272 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T08:46:26.273 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-27T08:46:26.285 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-27T08:46:26.286 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-27T08:46:26.301 DEBUG:teuthology.orchestra.run.smithi183:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-27T08:46:26.377 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-27T08:46:26.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T08:46:26.406 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-27T08:46:26.416 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-27T08:46:26.431 INFO:teuthology.task.internal:Duration was 593.722019 seconds 2024-03-27T08:46:26.431 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-27T08:46:26.446 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-27T08:46:26.446 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-27T08:46:26.489 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-27T08:46:26.864 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-27T08:46:26.864 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi183.front.sepia.ceph.com 2024-03-27T08:46:26.865 DEBUG:teuthology.orchestra.run.smithi183:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-27T08:46:26.924 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-27T08:46:26.924 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-27T08:46:26.992 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-27T08:46:26.993 DEBUG:teuthology.orchestra.run.smithi183:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-27T08:46:27.129 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-27T08:46:27.139 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-27T08:46:27.139 DEBUG:teuthology.orchestra.run.smithi183:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-27T08:46:27.165 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-27T08:46:27.174 DEBUG:teuthology.orchestra.run.smithi183:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-27T08:46:27.230 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = core 2024-03-27T08:46:27.246 DEBUG:teuthology.orchestra.run.smithi183:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-27T08:46:27.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T08:46:27.301 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-27T08:46:27.311 INFO:teuthology.task.internal:Transferring archived files... 2024-03-27T08:46:27.311 DEBUG:teuthology.misc:Transferring archived files from smithi183:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2024-03-27_08:05:55-crimson-rados-wip-matanb-crimson-foreign-ptr-distro-crimson-smithi/7625904/remote/smithi183 2024-03-27T08:46:27.311 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-27T08:46:27.375 INFO:teuthology.task.internal:Removing archive directory... 2024-03-27T08:46:27.375 DEBUG:teuthology.orchestra.run.smithi183:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-27T08:46:27.424 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-27T08:46:27.437 INFO:teuthology.task.internal:Not uploading archives. 2024-03-27T08:46:27.437 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-27T08:46:27.452 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-27T08:46:27.452 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-27T08:46:27.479 INFO:teuthology.orchestra.run.smithi183.stdout: 265594 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 08:46 /home/ubuntu/cephtest 2024-03-27T08:46:27.480 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-27T08:46:27.492 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-27T08:46:27.523 INFO:teuthology.nuke:Checking targets against current locks 2024-03-27T08:46:27.548 DEBUG:teuthology.nuke:shortname: smithi183 2024-03-27T08:46:27.548 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-27T08:46:27.571 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-03-27_08:05:55-crimson-rados-wip-matanb-crimson-foreign-ptr-distro-crimson-smithi/7625904', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-27 08:28:41.389800', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKX/Lm8n4pB52qJdUdfLbrooLRiRZrAIg+jORRiQmF+LaN66a4DgOu9uW2B6A2tbD9+uN1ol8rmfSmTXPcjkVyA='} 2024-03-27T08:46:27.623 INFO:teuthology.orchestra.console.smithi183:Power off 2024-03-27T08:46:27.623 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-27T08:46:27.646 DEBUG:teuthology.orchestra.console.smithi183:power off output: Chassis Power Control: Down/Off 2024-03-27T08:46:27.646 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T08:46:27.669 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-03-27T08:46:31.670 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T08:46:31.778 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-03-27T08:46:35.783 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T08:46:35.894 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is off 2024-03-27T08:46:35.995 INFO:teuthology.orchestra.console.smithi183:Power off completed 2024-03-27T08:46:36.099 INFO:teuthology.run:Summary data: description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_lock_and_fence} duration: 593.7220191955566 failure_reason: '"2024-03-27T08:46:06.843999+0000 mon.a (mon.0) 143 : cluster [ERR] Health check failed: 9 scrub errors (OSD_SCRUB_ERRORS)" in cluster log' flavor: crimson owner: scheduled_matan@teuthology success: false 2024-03-27T08:46:36.099 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-27T08:46:36.190 INFO:teuthology.run:FAIL