2024-09-16T16:29:19.554 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T16:29:19.573 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T16:29:19.641 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2024-09-16_16:23:48-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi/7907402 branch: wip-matanb-crimson-testing-59543 description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore tasks/rbd_lock_and_fence} email: null first_in_suite: false job_id: '7907402' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2024-09-16_16:23:48-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-matanb-crimson-testing-59543 ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mgr stats period: 30 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-1 osd.1: crimson seastar cpu cores: 2-3 osd.2: crimson seastar cpu cores: 4-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: 81fedcb3b484c843d22ffbc8d179cb567b7083b5 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: crimson sha1: 81fedcb3b484c843d22ffbc8d179cb567b7083b5 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 0325fd3c11a225c0fd9eb38fab88aad1370ad779 owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 6678 sha1: 81fedcb3b484c843d22ffbc8d179cb567b7083b5 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 0325fd3c11a225c0fd9eb38fab88aad1370ad779 targets: smithi026.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIaeFpgmyXprtzZw6b9E7XAb5is+fCrTfoIGxEOLdv0i2Tkp6JIygOg1w50yJ8ZO+JWzBMBwx0efv8JiJ7SEiA0= tasks: - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 flavor: crimson - workunit: clients: client.0: - rbd/test_lock_fence.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-16_16:23:48 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T16:29:19.641 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa; will attempt to use it 2024-09-16T16:29:19.642 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks 2024-09-16T16:29:19.642 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T16:29:19.644 INFO:teuthology.task.internal:Checking packages... 2024-09-16T16:29:19.676 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '81fedcb3b484c843d22ffbc8d179cb567b7083b5' 2024-09-16T16:29:19.676 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T16:29:19.676 INFO:teuthology.packaging:ref: None 2024-09-16T16:29:19.676 INFO:teuthology.packaging:tag: None 2024-09-16T16:29:19.676 INFO:teuthology.packaging:branch: wip-matanb-crimson-testing-59543 2024-09-16T16:29:19.676 INFO:teuthology.packaging:sha1: 81fedcb3b484c843d22ffbc8d179cb567b7083b5 2024-09-16T16:29:19.677 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-matanb-crimson-testing-59543 2024-09-16T16:29:19.827 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4879.g81fedcb3 2024-09-16T16:29:19.828 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T16:29:19.834 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T16:29:19.834 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T16:29:19.910 INFO:teuthology.task.internal:Saving configuration 2024-09-16T16:29:19.921 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T16:29:19.943 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T16:29:19.964 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi026.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-09-16_16:23:48-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi/7907402', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 16:25:02.785301', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIaeFpgmyXprtzZw6b9E7XAb5is+fCrTfoIGxEOLdv0i2Tkp6JIygOg1w50yJ8ZO+JWzBMBwx0efv8JiJ7SEiA0='} 2024-09-16T16:29:19.964 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T16:29:19.977 INFO:teuthology.task.internal:roles: ubuntu@smithi026.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-16T16:29:19.977 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T16:29:20.036 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f288ea57be0>, signals=[15]) 2024-09-16T16:29:20.036 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T16:29:20.043 INFO:teuthology.task.internal:Opening connections... 2024-09-16T16:29:20.044 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi026.front.sepia.ceph.com 2024-09-16T16:29:20.045 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T16:29:20.121 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T16:29:20.129 DEBUG:teuthology.orchestra.run.smithi026:> uname -m 2024-09-16T16:29:20.206 INFO:teuthology.orchestra.run.smithi026.stdout:x86_64 2024-09-16T16:29:20.206 DEBUG:teuthology.orchestra.run.smithi026:> cat /etc/os-release 2024-09-16T16:29:20.264 INFO:teuthology.orchestra.run.smithi026.stdout:NAME="CentOS Stream" 2024-09-16T16:29:20.264 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION="9" 2024-09-16T16:29:20.264 INFO:teuthology.orchestra.run.smithi026.stdout:ID="centos" 2024-09-16T16:29:20.264 INFO:teuthology.orchestra.run.smithi026.stdout:ID_LIKE="rhel fedora" 2024-09-16T16:29:20.264 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION_ID="9" 2024-09-16T16:29:20.264 INFO:teuthology.orchestra.run.smithi026.stdout:PLATFORM_ID="platform:el9" 2024-09-16T16:29:20.264 INFO:teuthology.orchestra.run.smithi026.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T16:29:20.264 INFO:teuthology.orchestra.run.smithi026.stdout:ANSI_COLOR="0;31" 2024-09-16T16:29:20.265 INFO:teuthology.orchestra.run.smithi026.stdout:LOGO="fedora-logo-icon" 2024-09-16T16:29:20.265 INFO:teuthology.orchestra.run.smithi026.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T16:29:20.265 INFO:teuthology.orchestra.run.smithi026.stdout:HOME_URL="https://centos.org/" 2024-09-16T16:29:20.265 INFO:teuthology.orchestra.run.smithi026.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T16:29:20.265 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T16:29:20.265 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T16:29:20.265 INFO:teuthology.lock.ops:Updating smithi026.front.sepia.ceph.com on lock server 2024-09-16T16:29:20.286 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T16:29:20.294 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T16:29:20.319 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T16:29:20.319 DEBUG:teuthology.orchestra.run.smithi026:> test '!' -e /home/ubuntu/cephtest 2024-09-16T16:29:20.336 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T16:29:20.352 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T16:29:20.352 DEBUG:teuthology.orchestra.run.smithi026:> test -z $(ls -A /var/lib/ceph) 2024-09-16T16:29:20.398 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T16:29:20.429 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T16:29:20.444 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-09-16T16:29:20.444 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-09-16T16:29:20.445 DEBUG:teuthology.orchestra.run.smithi026:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T16:29:20.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T16:29:20.461 DEBUG:teuthology.orchestra.run.smithi026:> uname -r 2024-09-16T16:29:20.517 INFO:teuthology.orchestra.run.smithi026.stdout:5.14.0-503.el9.x86_64 2024-09-16T16:29:20.518 INFO:teuthology.task.kernel:Running kernel on smithi026: 5.14.0-503.el9.x86_64 2024-09-16T16:29:20.518 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum install -y kernel 2024-09-16T16:29:22.909 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-09-16T16:29:27.376 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - AppStream 6.2 MB/s | 20 MB 00:03 2024-09-16T16:29:31.927 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.5 MB 00:00 2024-09-16T16:29:33.444 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - Extras packages 70 kB/s | 19 kB 00:00 2024-09-16T16:29:35.007 INFO:teuthology.orchestra.run.smithi026.stdout:Extra Packages for Enterprise Linux 16 MB/s | 23 MB 00:01 2024-09-16T16:29:40.696 INFO:teuthology.orchestra.run.smithi026.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-09-16T16:29:42.794 INFO:teuthology.orchestra.run.smithi026.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T16:29:42.794 INFO:teuthology.orchestra.run.smithi026.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T16:29:42.894 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-16T16:29:42.899 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:29:42.899 INFO:teuthology.orchestra.run.smithi026.stdout: Package Architecture Version Repository Size 2024-09-16T16:29:42.899 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:29:42.899 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-16T16:29:42.900 INFO:teuthology.orchestra.run.smithi026.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T16:29:42.900 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-09-16T16:29:42.900 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T16:29:42.900 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T16:29:42.900 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T16:29:42.900 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:29:42.900 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-16T16:29:42.900 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:29:42.900 INFO:teuthology.orchestra.run.smithi026.stdout:Install 4 Packages 2024-09-16T16:29:42.900 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:29:42.901 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 79 M 2024-09-16T16:29:42.901 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 126 M 2024-09-16T16:29:42.901 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-16T16:29:43.497 INFO:teuthology.orchestra.run.smithi026.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 40 kB/s | 23 kB 00:00 2024-09-16T16:29:44.356 INFO:teuthology.orchestra.run.smithi026.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-16T16:29:45.481 INFO:teuthology.orchestra.run.smithi026.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 13 MB/s | 35 MB 00:02 2024-09-16T16:29:45.964 INFO:teuthology.orchestra.run.smithi026.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 12 MB/s | 29 MB 00:02 2024-09-16T16:29:45.964 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:29:45.965 INFO:teuthology.orchestra.run.smithi026.stdout:Total 26 MB/s | 79 MB 00:03 2024-09-16T16:29:46.672 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-16T16:29:47.029 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-16T16:29:47.029 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-16T16:29:48.037 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-16T16:29:48.037 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-16T16:29:50.629 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-16T16:29:51.302 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T16:29:51.381 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T16:29:54.165 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T16:29:54.316 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T16:29:59.439 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T16:29:59.491 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T16:30:04.625 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T16:30:43.679 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T16:30:43.720 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T16:30:44.085 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T16:30:44.085 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T16:30:44.086 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T16:30:44.086 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T16:30:44.966 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T16:30:44.966 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:30:44.966 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-16T16:30:44.966 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T16:30:44.966 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T16:30:44.966 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T16:30:44.966 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T16:30:44.967 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:30:44.967 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-16T16:30:45.319 DEBUG:teuthology.orchestra.run.smithi026:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T16:30:45.410 INFO:teuthology.orchestra.run.smithi026.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T16:30:45.410 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T16:30:45.410 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T16:30:45.411 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T16:30:45.411 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T16:30:45.411 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T16:30:45.411 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi026.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T16:30:45.411 DEBUG:teuthology.orchestra.run.smithi026:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T16:30:45.470 INFO:teuthology.orchestra.run.smithi026.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T16:30:45.470 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T16:30:45.470 DEBUG:teuthology.orchestra.run.smithi026:> sudo rpm -qi grub2-tools 2024-09-16T16:30:45.502 INFO:teuthology.orchestra.run.smithi026.stdout:Name : grub2-tools 2024-09-16T16:30:45.502 INFO:teuthology.orchestra.run.smithi026.stdout:Epoch : 1 2024-09-16T16:30:45.502 INFO:teuthology.orchestra.run.smithi026.stdout:Version : 2.06 2024-09-16T16:30:45.502 INFO:teuthology.orchestra.run.smithi026.stdout:Release : 82.el9 2024-09-16T16:30:45.502 INFO:teuthology.orchestra.run.smithi026.stdout:Architecture: x86_64 2024-09-16T16:30:45.502 INFO:teuthology.orchestra.run.smithi026.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T16:30:45.502 INFO:teuthology.orchestra.run.smithi026.stdout:Group : Unspecified 2024-09-16T16:30:45.502 INFO:teuthology.orchestra.run.smithi026.stdout:Size : 8274670 2024-09-16T16:30:45.502 INFO:teuthology.orchestra.run.smithi026.stdout:License : GPLv3+ 2024-09-16T16:30:45.503 INFO:teuthology.orchestra.run.smithi026.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T16:30:45.503 INFO:teuthology.orchestra.run.smithi026.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T16:30:45.503 INFO:teuthology.orchestra.run.smithi026.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T16:30:45.503 INFO:teuthology.orchestra.run.smithi026.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T16:30:45.503 INFO:teuthology.orchestra.run.smithi026.stdout:Packager : builder@centos.org 2024-09-16T16:30:45.503 INFO:teuthology.orchestra.run.smithi026.stdout:Vendor : CentOS 2024-09-16T16:30:45.503 INFO:teuthology.orchestra.run.smithi026.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T16:30:45.503 INFO:teuthology.orchestra.run.smithi026.stdout:Summary : Support tools for GRUB. 2024-09-16T16:30:45.503 INFO:teuthology.orchestra.run.smithi026.stdout:Description : 2024-09-16T16:30:45.503 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:30:45.503 INFO:teuthology.orchestra.run.smithi026.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T16:30:45.503 INFO:teuthology.orchestra.run.smithi026.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T16:30:45.503 INFO:teuthology.orchestra.run.smithi026.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T16:30:45.504 INFO:teuthology.orchestra.run.smithi026.stdout:hardware devices. 2024-09-16T16:30:45.504 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:30:45.504 INFO:teuthology.orchestra.run.smithi026.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T16:30:45.504 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T16:30:45.505 INFO:teuthology.task.kernel:Updating grub on smithi026 to boot 5.14.0-505.el9.x86_64 2024-09-16T16:30:45.505 DEBUG:teuthology.orchestra.run.smithi026:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T16:30:46.213 INFO:teuthology.orchestra.run.smithi026.stderr:Generating grub configuration file ... 2024-09-16T16:30:47.619 INFO:teuthology.orchestra.run.smithi026.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T16:30:47.664 INFO:teuthology.orchestra.run.smithi026.stderr:done 2024-09-16T16:30:47.666 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2024-09-16T16:30:47.722 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.hvk4EpGwzg 2024-09-16T16:30:47.723 DEBUG:teuthology.orchestra.run.smithi026:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.hvk4EpGwzg 2024-09-16T16:30:47.794 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.hvk4EpGwzg 2024-09-16T16:30:47.944 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.hvk4EpGwzg is 6KB 2024-09-16T16:30:47.992 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.hvk4EpGwzg 2024-09-16T16:30:48.007 DEBUG:teuthology.orchestra.run.smithi026:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T16:30:48.070 INFO:teuthology.orchestra.run.smithi026.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T16:30:48.070 INFO:teuthology.orchestra.run.smithi026.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T16:30:48.070 INFO:teuthology.orchestra.run.smithi026.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T16:30:48.071 INFO:teuthology.orchestra.run.smithi026.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T16:30:48.072 DEBUG:teuthology.orchestra.run.smithi026:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T16:30:48.218 DEBUG:teuthology.orchestra.run.smithi026:> sudo shutdown -r now 2024-09-16T16:31:18.248 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T16:31:18.249 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-09-16T16:31:18.250 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T16:31:36.765 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.26 2024-09-16T16:31:45.773 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-09-16T16:31:45.773 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T16:32:04.157 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.26 2024-09-16T16:32:16.167 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-09-16T16:32:16.168 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T16:32:17.695 DEBUG:teuthology.orchestra.run.smithi026:> true 2024-09-16T16:32:18.108 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-09-16T16:32:18.108 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-16T16:32:18.109 DEBUG:teuthology.orchestra.run.smithi026:> uname -r 2024-09-16T16:32:18.123 INFO:teuthology.orchestra.run.smithi026.stdout:5.14.0-505.el9.x86_64 2024-09-16T16:32:18.124 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T16:32:18.124 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T16:32:18.124 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T16:32:19.125 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T16:32:19.125 DEBUG:teuthology.orchestra.run.smithi026:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T16:32:19.202 INFO:teuthology.orchestra.run.smithi026.stdout:ttyS1 2024-09-16T16:32:19.224 DEBUG:teuthology.parallel:result is None 2024-09-16T16:32:19.224 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T16:32:19.230 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T16:32:19.231 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T16:32:19.246 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T16:32:19.254 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T16:32:19.297 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T16:32:19.297 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T16:32:19.325 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T16:32:19.346 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T16:32:19.347 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T16:32:19.390 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:32:19.401 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T16:32:19.403 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T16:32:19.409 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T16:32:19.409 DEBUG:teuthology.orchestra.run.smithi026:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T16:32:19.469 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T16:32:19.478 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T16:32:19.478 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T16:32:19.525 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T16:32:19.607 DEBUG:teuthology.orchestra.run.smithi026:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T16:32:19.653 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T16:32:19.735 DEBUG:teuthology.orchestra.run.smithi026:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T16:32:19.776 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T16:32:19.776 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T16:32:19.849 DEBUG:teuthology.orchestra.run.smithi026:> sudo service rsyslog restart 2024-09-16T16:32:19.935 INFO:teuthology.orchestra.run.smithi026.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T16:32:20.329 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T16:32:20.339 INFO:teuthology.task.internal:Starting timer... 2024-09-16T16:32:20.339 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T16:32:20.376 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T16:32:20.425 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-16T16:32:20.426 DEBUG:teuthology.orchestra.run.smithi026:> sudo service auditd rotate 2024-09-16T16:32:20.468 INFO:teuthology.orchestra.run.smithi026.stdout:Rotating logs: 2024-09-16T16:32:20.470 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T16:32:20.470 DEBUG:teuthology.orchestra.run.smithi026:> /usr/sbin/getenforce 2024-09-16T16:32:20.545 INFO:teuthology.orchestra.run.smithi026.stdout:Permissive 2024-09-16T16:32:20.545 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi026.front.sepia.ceph.com': 'permissive'} 2024-09-16T16:32:20.546 DEBUG:teuthology.orchestra.run.smithi026:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T16:32:20.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T16:32:20.614 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T16:32:20.615 DEBUG:teuthology.orchestra.run.smithi026:> sudo /usr/sbin/setenforce permissive 2024-09-16T16:32:20.687 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T16:32:20.701 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-16T16:32:20.704 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-16T16:32:20.867 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T16:32:20.878 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T16:32:20.879 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi026.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T16:36:20.565 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi026.front.sepia.ceph.com')] 2024-09-16T16:36:20.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-09-16T16:36:20.568 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T16:36:20.701 DEBUG:teuthology.orchestra.run.smithi026:> true 2024-09-16T16:36:20.785 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-09-16T16:36:20.785 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T16:36:20.799 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T16:36:20.799 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T16:36:20.800 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T16:36:20.855 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T16:36:20.870 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T16:36:20.896 INFO:teuthology.orchestra.run.smithi026.stderr:sudo: ntpd: command not found 2024-09-16T16:36:20.908 INFO:teuthology.orchestra.run.smithi026.stdout:506 Cannot talk to daemon 2024-09-16T16:36:20.923 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T16:36:20.937 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T16:36:20.999 INFO:teuthology.orchestra.run.smithi026.stderr:bash: line 1: ntpq: command not found 2024-09-16T16:36:21.002 INFO:teuthology.orchestra.run.smithi026.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T16:36:21.002 INFO:teuthology.orchestra.run.smithi026.stdout:=============================================================================== 2024-09-16T16:36:21.002 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T16:36:21.002 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T16:36:21.002 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T16:36:21.002 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T16:36:21.003 INFO:teuthology.run_tasks:Running task install... 2024-09-16T16:36:21.012 DEBUG:teuthology.task.install:project ceph 2024-09-16T16:36:21.012 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '81fedcb3b484c843d22ffbc8d179cb567b7083b5'}} 2024-09-16T16:36:21.012 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '81fedcb3b484c843d22ffbc8d179cb567b7083b5'} 2024-09-16T16:36:21.012 INFO:teuthology.task.install:Using flavor: crimson 2024-09-16T16:36:21.018 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T16:36:21.018 INFO:teuthology.task.install:extra packages: [] 2024-09-16T16:36:21.021 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81fedcb3b484c843d22ffbc8d179cb567b7083b5', 'tag': None, 'wait_for_package': False} 2024-09-16T16:36:21.021 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=81fedcb3b484c843d22ffbc8d179cb567b7083b5 2024-09-16T16:36:21.174 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-59543/81fedcb3b484c843d22ffbc8d179cb567b7083b5/centos/9/flavors/crimson/ 2024-09-16T16:36:21.174 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4879.g81fedcb3 2024-09-16T16:36:21.293 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-59543/81fedcb3b484c843d22ffbc8d179cb567b7083b5/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-59543/81fedcb3b484c843d22ffbc8d179cb567b7083b5/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-59543/81fedcb3b484c843d22ffbc8d179cb567b7083b5/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T16:36:21.293 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T16:36:21.293 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T16:36:21.321 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T16:36:21.322 DEBUG:teuthology.orchestra.run.smithi026:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/81fedcb3b484c843d22ffbc8d179cb567b7083b5/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T16:36:21.388 DEBUG:teuthology.orchestra.run.smithi026:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T16:36:21.465 DEBUG:teuthology.orchestra.run.smithi026:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T16:36:21.530 INFO:teuthology.orchestra.run.smithi026.stdout:check_obsoletes = 1 2024-09-16T16:36:21.531 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum clean all 2024-09-16T16:36:21.772 INFO:teuthology.orchestra.run.smithi026.stdout:45 files removed 2024-09-16T16:36:21.799 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-radosgw 2024-09-16T16:36:22.381 INFO:teuthology.orchestra.run.smithi026.stdout:ceph packages for x86_64 318 kB/s | 88 kB 00:00 2024-09-16T16:36:22.672 INFO:teuthology.orchestra.run.smithi026.stdout:ceph noarch packages 98 kB/s | 24 kB 00:00 2024-09-16T16:36:22.945 INFO:teuthology.orchestra.run.smithi026.stdout:ceph source packages 8.8 kB/s | 2.1 kB 00:00 2024-09-16T16:36:23.695 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-09-16T16:36:29.254 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - AppStream 4.6 MB/s | 20 MB 00:04 2024-09-16T16:36:34.330 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - CRB 6.2 MB/s | 6.5 MB 00:01 2024-09-16T16:36:35.856 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - Extras packages 81 kB/s | 19 kB 00:00 2024-09-16T16:36:37.110 INFO:teuthology.orchestra.run.smithi026.stdout:Extra Packages for Enterprise Linux 20 MB/s | 23 MB 00:01 2024-09-16T16:36:42.825 INFO:teuthology.orchestra.run.smithi026.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-16T16:36:45.278 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-16T16:36:45.290 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:36:45.291 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-09-16T16:36:45.291 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:36:45.291 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-16T16:36:45.291 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-radosgw x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 10 M 2024-09-16T16:36:45.291 INFO:teuthology.orchestra.run.smithi026.stdout:Upgrading: 2024-09-16T16:36:45.291 INFO:teuthology.orchestra.run.smithi026.stdout: librados2 x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 3.5 M 2024-09-16T16:36:45.291 INFO:teuthology.orchestra.run.smithi026.stdout: librbd1 x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 3.8 M 2024-09-16T16:36:45.291 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-09-16T16:36:45.291 INFO:teuthology.orchestra.run.smithi026.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T16:36:45.291 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-base x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 6.1 M 2024-09-16T16:36:45.291 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-common x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 20 M 2024-09-16T16:36:45.291 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-selinux x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 25 k 2024-09-16T16:36:45.291 INFO:teuthology.orchestra.run.smithi026.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T16:36:45.292 INFO:teuthology.orchestra.run.smithi026.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T16:36:45.292 INFO:teuthology.orchestra.run.smithi026.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T16:36:45.292 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs2 x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 809 k 2024-09-16T16:36:45.292 INFO:teuthology.orchestra.run.smithi026.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-16T16:36:45.292 INFO:teuthology.orchestra.run.smithi026.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T16:36:45.292 INFO:teuthology.orchestra.run.smithi026.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T16:36:45.292 INFO:teuthology.orchestra.run.smithi026.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T16:36:45.292 INFO:teuthology.orchestra.run.smithi026.stdout: libradosstriper1 x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 558 k 2024-09-16T16:36:45.292 INFO:teuthology.orchestra.run.smithi026.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T16:36:45.292 INFO:teuthology.orchestra.run.smithi026.stdout: librgw2 x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 5.4 M 2024-09-16T16:36:45.292 INFO:teuthology.orchestra.run.smithi026.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T16:36:45.292 INFO:teuthology.orchestra.run.smithi026.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T16:36:45.292 INFO:teuthology.orchestra.run.smithi026.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T16:36:45.293 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-argparse x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 45 k 2024-09-16T16:36:45.293 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-common x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 143 k 2024-09-16T16:36:45.293 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cephfs x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 160 k 2024-09-16T16:36:45.293 INFO:teuthology.orchestra.run.smithi026.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T16:36:45.293 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rados x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 315 k 2024-09-16T16:36:45.293 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rbd x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 302 k 2024-09-16T16:36:45.293 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rgw x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 99 k 2024-09-16T16:36:45.293 INFO:teuthology.orchestra.run.smithi026.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T16:36:45.293 INFO:teuthology.orchestra.run.smithi026.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T16:36:45.293 INFO:teuthology.orchestra.run.smithi026.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T16:36:45.293 INFO:teuthology.orchestra.run.smithi026.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T16:36:45.293 INFO:teuthology.orchestra.run.smithi026.stdout:Installing weak dependencies: 2024-09-16T16:36:45.293 INFO:teuthology.orchestra.run.smithi026.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T16:36:45.294 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:36:45.294 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-16T16:36:45.294 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:36:45.294 INFO:teuthology.orchestra.run.smithi026.stdout:Install 31 Packages 2024-09-16T16:36:45.294 INFO:teuthology.orchestra.run.smithi026.stdout:Upgrade 2 Packages 2024-09-16T16:36:45.294 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:36:45.295 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 60 M 2024-09-16T16:36:45.295 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-16T16:36:46.459 INFO:teuthology.orchestra.run.smithi026.stdout:(1/33): ceph-radosgw-19.3.0-4879.g81fedcb3.el9. 11 MB/s | 10 MB 00:00 2024-09-16T16:36:46.525 INFO:teuthology.orchestra.run.smithi026.stdout:(2/33): ceph-selinux-19.3.0-4879.g81fedcb3.el9. 378 kB/s | 25 kB 00:00 2024-09-16T16:36:46.667 INFO:teuthology.orchestra.run.smithi026.stdout:(3/33): libcephfs2-19.3.0-4879.g81fedcb3.el9.x8 5.6 MB/s | 809 kB 00:00 2024-09-16T16:36:46.784 INFO:teuthology.orchestra.run.smithi026.stdout:(4/33): libradosstriper1-19.3.0-4879.g81fedcb3. 4.7 MB/s | 558 kB 00:00 2024-09-16T16:36:47.209 INFO:teuthology.orchestra.run.smithi026.stdout:(5/33): librgw2-19.3.0-4879.g81fedcb3.el9.x86_6 13 MB/s | 5.4 MB 00:00 2024-09-16T16:36:47.451 INFO:teuthology.orchestra.run.smithi026.stdout:(6/33): ceph-common-19.3.0-4879.g81fedcb3.el9.x 10 MB/s | 20 MB 00:01 2024-09-16T16:36:47.476 INFO:teuthology.orchestra.run.smithi026.stdout:(7/33): python3-ceph-argparse-19.3.0-4879.g81fe 169 kB/s | 45 kB 00:00 2024-09-16T16:36:47.560 INFO:teuthology.orchestra.run.smithi026.stdout:(8/33): ceph-base-19.3.0-4879.g81fedcb3.el9.x86 2.9 MB/s | 6.1 MB 00:02 2024-09-16T16:36:47.589 INFO:teuthology.orchestra.run.smithi026.stdout:(9/33): python3-ceph-common-19.3.0-4879.g81fedc 1.0 MB/s | 143 kB 00:00 2024-09-16T16:36:47.611 INFO:teuthology.orchestra.run.smithi026.stdout:(10/33): python3-cephfs-19.3.0-4879.g81fedcb3.e 1.2 MB/s | 160 kB 00:00 2024-09-16T16:36:47.669 INFO:teuthology.orchestra.run.smithi026.stdout:(11/33): python3-rados-19.3.0-4879.g81fedcb3.el 2.8 MB/s | 315 kB 00:00 2024-09-16T16:36:47.697 INFO:teuthology.orchestra.run.smithi026.stdout:(12/33): python3-rgw-19.3.0-4879.g81fedcb3.el9. 1.2 MB/s | 99 kB 00:00 2024-09-16T16:36:47.720 INFO:teuthology.orchestra.run.smithi026.stdout:(13/33): python3-rbd-19.3.0-4879.g81fedcb3.el9. 2.2 MB/s | 302 kB 00:00 2024-09-16T16:36:47.862 INFO:teuthology.orchestra.run.smithi026.stdout:(14/33): libnbd-1.20.2-2.el9.x86_64.rpm 1.0 MB/s | 171 kB 00:00 2024-09-16T16:36:47.895 INFO:teuthology.orchestra.run.smithi026.stdout:(15/33): boost-program-options-1.75.0-8.el9.x86 473 kB/s | 107 kB 00:00 2024-09-16T16:36:47.921 INFO:teuthology.orchestra.run.smithi026.stdout:(16/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 775 kB/s | 45 kB 00:00 2024-09-16T16:36:47.958 INFO:teuthology.orchestra.run.smithi026.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 684 kB/s | 160 kB 00:00 2024-09-16T16:36:48.013 INFO:teuthology.orchestra.run.smithi026.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 727 kB/s | 42 kB 00:00 2024-09-16T16:36:48.057 INFO:teuthology.orchestra.run.smithi026.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-09-16T16:36:48.084 INFO:teuthology.orchestra.run.smithi026.stdout:(20/33): librdkafka-1.6.1-102.el9.x86_64.rpm 3.5 MB/s | 662 kB 00:00 2024-09-16T16:36:48.108 INFO:teuthology.orchestra.run.smithi026.stdout:(21/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.3 MB/s | 221 kB 00:00 2024-09-16T16:36:48.135 INFO:teuthology.orchestra.run.smithi026.stdout:(22/33): qatlib-service-24.02.0-1.el9.x86_64.rp 475 kB/s | 36 kB 00:00 2024-09-16T16:36:48.159 INFO:teuthology.orchestra.run.smithi026.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 627 kB/s | 47 kB 00:00 2024-09-16T16:36:48.225 INFO:teuthology.orchestra.run.smithi026.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.6 MB/s | 308 kB 00:00 2024-09-16T16:36:48.250 INFO:teuthology.orchestra.run.smithi026.stdout:(25/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 345 kB/s | 26 kB 00:00 2024-09-16T16:36:48.276 INFO:teuthology.orchestra.run.smithi026.stdout:(26/33): liboath-2.6.7-2.el9.x86_64.rpm 968 kB/s | 49 kB 00:00 2024-09-16T16:36:48.301 INFO:teuthology.orchestra.run.smithi026.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-16T16:36:48.343 INFO:teuthology.orchestra.run.smithi026.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-16T16:36:48.385 INFO:teuthology.orchestra.run.smithi026.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-09-16T16:36:48.485 INFO:teuthology.orchestra.run.smithi026.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-09-16T16:36:48.577 INFO:teuthology.orchestra.run.smithi026.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 9.9 MB/s | 4.4 MB 00:00 2024-09-16T16:36:48.752 INFO:teuthology.orchestra.run.smithi026.stdout:(32/33): librbd1-19.3.0-4879.g81fedcb3.el9.x86_ 14 MB/s | 3.8 MB 00:00 2024-09-16T16:36:48.886 INFO:teuthology.orchestra.run.smithi026.stdout:(33/33): librados2-19.3.0-4879.g81fedcb3.el9.x8 6.9 MB/s | 3.5 MB 00:00 2024-09-16T16:36:48.891 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:36:48.891 INFO:teuthology.orchestra.run.smithi026.stdout:Total 17 MB/s | 60 MB 00:03 2024-09-16T16:36:49.009 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-16T16:36:49.050 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-16T16:36:49.050 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-16T16:36:49.756 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-16T16:36:49.757 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-16T16:36:51.297 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-16T16:36:51.866 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-16T16:36:51.921 INFO:teuthology.orchestra.run.smithi026.stdout: Upgrading : librados2-2:19.3.0-4879.g81fedcb3.el9.x86_64 2/35 2024-09-16T16:36:52.006 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados2-2:19.3.0-4879.g81fedcb3.el9.x86_64 2/35 2024-09-16T16:36:52.102 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rados-2:19.3.0-4879.g81fedcb3.el9.x86_64 3/35 2024-09-16T16:36:52.133 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephfs2-2:19.3.0-4879.g81fedcb3.el9.x86_64 4/35 2024-09-16T16:36:52.194 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephfs2-2:19.3.0-4879.g81fedcb3.el9.x86_64 4/35 2024-09-16T16:36:52.240 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/35 2024-09-16T16:36:52.269 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/35 2024-09-16T16:36:52.295 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/35 2024-09-16T16:36:52.333 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-16T16:36:52.364 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-16T16:36:52.513 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-16T16:36:52.551 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-16T16:36:52.585 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/35 2024-09-16T16:36:52.629 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-ceph-argparse-2:19.3.0-4879.g81fedcb3.el9. 11/35 2024-09-16T16:36:52.658 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cephfs-2:19.3.0-4879.g81fedcb3.el9.x86_64 12/35 2024-09-16T16:36:52.748 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 13/35 2024-09-16T16:36:52.771 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libradosstriper1-2:19.3.0-4879.g81fedcb3.el9.x86_6 14/35 2024-09-16T16:36:52.842 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4879.g81fedcb3.el9.x86_6 14/35 2024-09-16T16:36:52.878 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : thrift-0.15.0-2.el9.x86_64 15/35 2024-09-16T16:36:52.991 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : re2-1:20211101-20.el9.x86_64 16/35 2024-09-16T16:36:53.024 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 17/35 2024-09-16T16:36:53.072 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 18/35 2024-09-16T16:36:53.103 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 19/35 2024-09-16T16:36:53.133 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 20/35 2024-09-16T16:36:53.169 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 21/35 2024-09-16T16:36:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-16T16:36:53.604 INFO:teuthology.orchestra.run.smithi026.stdout: Upgrading : librbd1-2:19.3.0-4879.g81fedcb3.el9.x86_64 23/35 2024-09-16T16:36:53.695 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librbd1-2:19.3.0-4879.g81fedcb3.el9.x86_64 23/35 2024-09-16T16:36:53.726 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 24/35 2024-09-16T16:36:53.766 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-16T16:36:54.385 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-16T16:36:54.414 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librgw2-2:19.3.0-4879.g81fedcb3.el9.x86_64 27/35 2024-09-16T16:36:54.497 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librgw2-2:19.3.0-4879.g81fedcb3.el9.x86_64 27/35 2024-09-16T16:36:54.579 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 28/35 2024-09-16T16:36:54.600 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-ceph-common-2:19.3.0-4879.g81fedcb3.el9.x8 29/35 2024-09-16T16:36:57.197 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-common-2:19.3.0-4879.g81fedcb3.el9.x86_64 30/35 2024-09-16T16:36:57.239 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-common-2:19.3.0-4879.g81fedcb3.el9.x86_64 30/35 2024-09-16T16:36:58.044 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-common-2:19.3.0-4879.g81fedcb3.el9.x86_64 30/35 2024-09-16T16:36:58.083 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-base-2:19.3.0-4879.g81fedcb3.el9.x86_64 31/35 2024-09-16T16:36:58.199 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-base-2:19.3.0-4879.g81fedcb3.el9.x86_64 31/35 2024-09-16T16:36:58.199 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T16:36:58.199 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T16:36:58.199 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:36:58.252 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-selinux-2:19.3.0-4879.g81fedcb3.el9.x86_64 32/35 2024-09-16T16:37:09.077 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4879.g81fedcb3.el9.x86_64 32/35 2024-09-16T16:37:09.077 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /sys 2024-09-16T16:37:09.077 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /proc 2024-09-16T16:37:09.077 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /mnt 2024-09-16T16:37:09.077 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /var/tmp 2024-09-16T16:37:09.077 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /home 2024-09-16T16:37:09.077 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /root 2024-09-16T16:37:09.077 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /tmp 2024-09-16T16:37:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:37:10.226 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-radosgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 33/35 2024-09-16T16:37:10.253 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 33/35 2024-09-16T16:37:10.253 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T16:37:10.253 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T16:37:10.253 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T16:37:10.253 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T16:37:10.254 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:37:10.255 INFO:teuthology.orchestra.run.smithi026.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-16T16:37:10.346 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-16T16:37:10.346 INFO:teuthology.orchestra.run.smithi026.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T16:37:11.723 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T16:37:11.723 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-base-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/35 2024-09-16T16:37:11.723 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-common-2:19.3.0-4879.g81fedcb3.el9.x86_64 2/35 2024-09-16T16:37:11.724 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-radosgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 3/35 2024-09-16T16:37:11.724 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-selinux-2:19.3.0-4879.g81fedcb3.el9.x86_64 4/35 2024-09-16T16:37:11.724 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephfs2-2:19.3.0-4879.g81fedcb3.el9.x86_64 5/35 2024-09-16T16:37:11.724 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libradosstriper1-2:19.3.0-4879.g81fedcb3.el9.x86_6 6/35 2024-09-16T16:37:11.724 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librgw2-2:19.3.0-4879.g81fedcb3.el9.x86_64 7/35 2024-09-16T16:37:11.724 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4879.g81fedcb3.el9. 8/35 2024-09-16T16:37:11.724 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-ceph-common-2:19.3.0-4879.g81fedcb3.el9.x8 9/35 2024-09-16T16:37:11.724 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cephfs-2:19.3.0-4879.g81fedcb3.el9.x86_64 10/35 2024-09-16T16:37:11.724 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rados-2:19.3.0-4879.g81fedcb3.el9.x86_64 11/35 2024-09-16T16:37:11.724 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 12/35 2024-09-16T16:37:11.724 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 13/35 2024-09-16T16:37:11.725 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-16T16:37:11.725 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-16T16:37:11.725 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-16T16:37:11.725 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-16T16:37:11.725 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-16T16:37:11.725 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-16T16:37:11.725 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-16T16:37:11.725 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-16T16:37:11.725 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-16T16:37:11.725 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-16T16:37:11.725 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-16T16:37:11.726 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-16T16:37:11.726 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-16T16:37:11.726 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-16T16:37:11.728 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-16T16:37:11.729 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-16T16:37:11.729 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-16T16:37:11.729 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-16T16:37:11.729 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados2-2:19.3.0-4879.g81fedcb3.el9.x86_64 32/35 2024-09-16T16:37:11.729 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-16T16:37:11.729 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librbd1-2:19.3.0-4879.g81fedcb3.el9.x86_64 34/35 2024-09-16T16:37:12.837 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T16:37:12.837 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:37:12.837 INFO:teuthology.orchestra.run.smithi026.stdout:Upgraded: 2024-09-16T16:37:12.837 INFO:teuthology.orchestra.run.smithi026.stdout: librados2-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:37:12.837 INFO:teuthology.orchestra.run.smithi026.stdout: librbd1-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:37:12.837 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-16T16:37:12.837 INFO:teuthology.orchestra.run.smithi026.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T16:37:12.838 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-base-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:37:12.838 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-common-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:37:12.838 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-radosgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:37:12.838 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-selinux-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:37:12.838 INFO:teuthology.orchestra.run.smithi026.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T16:37:12.838 INFO:teuthology.orchestra.run.smithi026.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T16:37:12.838 INFO:teuthology.orchestra.run.smithi026.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T16:37:12.838 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs2-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:37:12.838 INFO:teuthology.orchestra.run.smithi026.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-16T16:37:12.838 INFO:teuthology.orchestra.run.smithi026.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T16:37:12.838 INFO:teuthology.orchestra.run.smithi026.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T16:37:12.838 INFO:teuthology.orchestra.run.smithi026.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T16:37:12.839 INFO:teuthology.orchestra.run.smithi026.stdout: libradosstriper1-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:37:12.839 INFO:teuthology.orchestra.run.smithi026.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T16:37:12.839 INFO:teuthology.orchestra.run.smithi026.stdout: librgw2-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:37:12.839 INFO:teuthology.orchestra.run.smithi026.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T16:37:12.839 INFO:teuthology.orchestra.run.smithi026.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T16:37:12.839 INFO:teuthology.orchestra.run.smithi026.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T16:37:12.839 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-argparse-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:37:12.839 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-common-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:37:12.839 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cephfs-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:37:12.839 INFO:teuthology.orchestra.run.smithi026.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T16:37:12.840 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rados-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:37:12.840 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:37:12.840 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:37:12.841 INFO:teuthology.orchestra.run.smithi026.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T16:37:12.841 INFO:teuthology.orchestra.run.smithi026.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T16:37:12.841 INFO:teuthology.orchestra.run.smithi026.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T16:37:12.841 INFO:teuthology.orchestra.run.smithi026.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T16:37:12.841 INFO:teuthology.orchestra.run.smithi026.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T16:37:12.841 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:37:12.841 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-16T16:37:13.170 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-test 2024-09-16T16:37:13.757 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:31 ago on Mon 16 Sep 2024 04:36:42 PM UTC. 2024-09-16T16:37:13.904 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-16T16:37:13.904 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:37:13.904 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-09-16T16:37:13.904 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:37:13.904 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-16T16:37:13.904 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-test x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 85 M 2024-09-16T16:37:13.904 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-09-16T16:37:13.904 INFO:teuthology.orchestra.run.smithi026.stdout: libasan8 x86_64 14.2.1-1.2.el9 appstream 513 k 2024-09-16T16:37:13.905 INFO:teuthology.orchestra.run.smithi026.stdout: libcephsqlite x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 175 k 2024-09-16T16:37:13.905 INFO:teuthology.orchestra.run.smithi026.stdout: libubsan x86_64 11.5.0-2.el9 appstream 183 k 2024-09-16T16:37:13.905 INFO:teuthology.orchestra.run.smithi026.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-09-16T16:37:13.905 INFO:teuthology.orchestra.run.smithi026.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T16:37:13.905 INFO:teuthology.orchestra.run.smithi026.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T16:37:13.905 INFO:teuthology.orchestra.run.smithi026.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T16:37:13.905 INFO:teuthology.orchestra.run.smithi026.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-09-16T16:37:13.905 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:37:13.905 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-16T16:37:13.905 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:37:13.905 INFO:teuthology.orchestra.run.smithi026.stdout:Install 9 Packages 2024-09-16T16:37:13.905 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:37:13.905 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 87 M 2024-09-16T16:37:13.905 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 797 M 2024-09-16T16:37:13.906 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-16T16:37:14.209 INFO:teuthology.orchestra.run.smithi026.stdout:(1/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 769 kB/s | 94 kB 00:00 2024-09-16T16:37:14.317 INFO:teuthology.orchestra.run.smithi026.stdout:(2/9): libcephsqlite-19.3.0-4879.g81fedcb3.el9. 757 kB/s | 175 kB 00:00 2024-09-16T16:37:14.463 INFO:teuthology.orchestra.run.smithi026.stdout:(3/9): libasan8-14.2.1-1.2.el9.x86_64.rpm 2.0 MB/s | 513 kB 00:00 2024-09-16T16:37:14.493 INFO:teuthology.orchestra.run.smithi026.stdout:(4/9): libubsan-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 183 kB 00:00 2024-09-16T16:37:14.560 INFO:teuthology.orchestra.run.smithi026.stdout:(5/9): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-09-16T16:37:14.610 INFO:teuthology.orchestra.run.smithi026.stdout:(6/9): socat-1.7.4.1-6.el9.x86_64.rpm 2.5 MB/s | 304 kB 00:00 2024-09-16T16:37:14.627 INFO:teuthology.orchestra.run.smithi026.stdout:(7/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 953 kB/s | 64 kB 00:00 2024-09-16T16:37:14.677 INFO:teuthology.orchestra.run.smithi026.stdout:(8/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 1.8 MB/s | 122 kB 00:00 2024-09-16T16:37:16.594 INFO:teuthology.orchestra.run.smithi026.stdout:(9/9): ceph-test-19.3.0-4879.g81fedcb3.el9.x86_ 34 MB/s | 85 MB 00:02 2024-09-16T16:37:16.598 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:37:16.599 INFO:teuthology.orchestra.run.smithi026.stdout:Total 32 MB/s | 87 MB 00:02 2024-09-16T16:37:16.689 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-16T16:37:16.713 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-16T16:37:16.713 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-16T16:37:17.361 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-16T16:37:17.361 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-16T16:37:18.283 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-16T16:37:18.352 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-09-16T16:37:18.418 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-09-16T16:37:18.469 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/9 2024-09-16T16:37:18.494 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-09-16T16:37:18.524 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-16T16:37:18.548 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-16T16:37:18.649 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-16T16:37:18.693 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-09-16T16:37:18.725 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephsqlite-2:19.3.0-4879.g81fedcb3.el9.x86_64 8/9 2024-09-16T16:37:30.707 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4879.g81fedcb3.el9.x86_64 8/9 2024-09-16T16:37:30.793 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-test-2:19.3.0-4879.g81fedcb3.el9.x86_64 9/9 2024-09-16T16:37:37.104 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-test-2:19.3.0-4879.g81fedcb3.el9.x86_64 9/9 2024-09-16T16:37:37.105 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-test-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/9 2024-09-16T16:37:37.105 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephsqlite-2:19.3.0-4879.g81fedcb3.el9.x86_64 2/9 2024-09-16T16:37:37.105 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-09-16T16:37:37.105 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libasan8-14.2.1-1.2.el9.x86_64 4/9 2024-09-16T16:37:37.105 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-16T16:37:37.105 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-09-16T16:37:37.105 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 7/9 2024-09-16T16:37:37.105 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-09-16T16:37:37.520 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-09-16T16:37:37.521 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:37:37.521 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-16T16:37:37.521 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-test-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:37:37.521 INFO:teuthology.orchestra.run.smithi026.stdout: libasan8-14.2.1-1.2.el9.x86_64 2024-09-16T16:37:37.521 INFO:teuthology.orchestra.run.smithi026.stdout: libcephsqlite-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:37:37.521 INFO:teuthology.orchestra.run.smithi026.stdout: libubsan-11.5.0-2.el9.x86_64 2024-09-16T16:37:37.521 INFO:teuthology.orchestra.run.smithi026.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-09-16T16:37:37.521 INFO:teuthology.orchestra.run.smithi026.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T16:37:37.521 INFO:teuthology.orchestra.run.smithi026.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T16:37:37.521 INFO:teuthology.orchestra.run.smithi026.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T16:37:37.521 INFO:teuthology.orchestra.run.smithi026.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-09-16T16:37:37.521 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:37:37.522 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-16T16:37:37.864 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph 2024-09-16T16:37:38.451 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:56 ago on Mon 16 Sep 2024 04:36:42 PM UTC. 2024-09-16T16:37:38.592 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-16T16:37:38.595 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:37:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-09-16T16:37:38.595 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:37:38.595 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-16T16:37:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: ceph x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 6.5 k 2024-09-16T16:37:38.595 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-09-16T16:37:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mds x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 2.2 M 2024-09-16T16:37:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 1.6 M 2024-09-16T16:37:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 291 k 2024-09-16T16:37:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mon x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 5.1 M 2024-09-16T16:37:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-osd x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 70 M 2024-09-16T16:37:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T16:37:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T16:37:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T16:37:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T16:37:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T16:37:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T16:37:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T16:37:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T16:37:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T16:37:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T16:37:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-collections 2024-09-16T16:37:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-16T16:37:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T16:37:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T16:37:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T16:37:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T16:37:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T16:37:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T16:37:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T16:37:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T16:37:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T16:37:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T16:37:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T16:37:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T16:37:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T16:37:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T16:37:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T16:37:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T16:37:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T16:37:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T16:37:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T16:37:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T16:37:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:37:38.599 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-16T16:37:38.599 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:37:38.599 INFO:teuthology.orchestra.run.smithi026.stdout:Install 37 Packages 2024-09-16T16:37:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:37:38.599 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 84 M 2024-09-16T16:37:38.600 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 897 M 2024-09-16T16:37:38.600 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-16T16:37:38.961 INFO:teuthology.orchestra.run.smithi026.stdout:(1/37): ceph-19.3.0-4879.g81fedcb3.el9.x86_64.r 50 kB/s | 6.5 kB 00:00 2024-09-16T16:37:39.152 INFO:teuthology.orchestra.run.smithi026.stdout:(2/37): ceph-mds-19.3.0-4879.g81fedcb3.el9.x86_ 6.8 MB/s | 2.2 MB 00:00 2024-09-16T16:37:39.194 INFO:teuthology.orchestra.run.smithi026.stdout:(3/37): ceph-mgr-19.3.0-4879.g81fedcb3.el9.x86_ 4.4 MB/s | 1.6 MB 00:00 2024-09-16T16:37:39.286 INFO:teuthology.orchestra.run.smithi026.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4879.g81fe 3.1 MB/s | 291 kB 00:00 2024-09-16T16:37:39.487 INFO:teuthology.orchestra.run.smithi026.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-09-16T16:37:39.587 INFO:teuthology.orchestra.run.smithi026.stdout:(6/37): ceph-mon-19.3.0-4879.g81fedcb3.el9.x86_ 8.2 MB/s | 5.1 MB 00:00 2024-09-16T16:37:39.679 INFO:teuthology.orchestra.run.smithi026.stdout:(7/37): python3-cryptography-36.0.1-4.el9.x86_6 6.4 MB/s | 1.2 MB 00:00 2024-09-16T16:37:39.738 INFO:teuthology.orchestra.run.smithi026.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 2.3 MB/s | 135 kB 00:00 2024-09-16T16:37:39.780 INFO:teuthology.orchestra.run.smithi026.stdout:(9/37): python3-ply-3.11-14.el9.noarch.rpm 554 kB/s | 106 kB 00:00 2024-09-16T16:37:39.805 INFO:teuthology.orchestra.run.smithi026.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 1.8 MB/s | 125 kB 00:00 2024-09-16T16:37:39.888 INFO:teuthology.orchestra.run.smithi026.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.9 MB/s | 215 kB 00:00 2024-09-16T16:37:39.930 INFO:teuthology.orchestra.run.smithi026.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 188 kB 00:00 2024-09-16T16:37:39.972 INFO:teuthology.orchestra.run.smithi026.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 838 kB/s | 35 kB 00:00 2024-09-16T16:37:40.014 INFO:teuthology.orchestra.run.smithi026.stdout:(14/37): python3-packaging-20.9-5.el9.noarch.rp 1.8 MB/s | 77 kB 00:00 2024-09-16T16:37:40.040 INFO:teuthology.orchestra.run.smithi026.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-09-16T16:37:40.065 INFO:teuthology.orchestra.run.smithi026.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-16T16:37:40.091 INFO:teuthology.orchestra.run.smithi026.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 831 kB/s | 42 kB 00:00 2024-09-16T16:37:40.116 INFO:teuthology.orchestra.run.smithi026.stdout:(18/37): lua-devel-5.4.4-4.el9.x86_64.rpm 443 kB/s | 22 kB 00:00 2024-09-16T16:37:40.166 INFO:teuthology.orchestra.run.smithi026.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-09-16T16:37:40.191 INFO:teuthology.orchestra.run.smithi026.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 578 kB/s | 43 kB 00:00 2024-09-16T16:37:40.216 INFO:teuthology.orchestra.run.smithi026.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-09-16T16:37:40.241 INFO:teuthology.orchestra.run.smithi026.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 428 kB/s | 11 kB 00:00 2024-09-16T16:37:40.267 INFO:teuthology.orchestra.run.smithi026.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 704 kB/s | 18 kB 00:00 2024-09-16T16:37:40.292 INFO:teuthology.orchestra.run.smithi026.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 932 kB/s | 23 kB 00:00 2024-09-16T16:37:40.327 INFO:teuthology.orchestra.run.smithi026.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 774 kB/s | 19 kB 00:00 2024-09-16T16:37:40.343 INFO:teuthology.orchestra.run.smithi026.stdout:(26/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.3 MB/s | 358 kB 00:00 2024-09-16T16:37:40.368 INFO:teuthology.orchestra.run.smithi026.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-16T16:37:40.394 INFO:teuthology.orchestra.run.smithi026.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 920 kB/s | 46 kB 00:00 2024-09-16T16:37:40.419 INFO:teuthology.orchestra.run.smithi026.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-16T16:37:40.444 INFO:teuthology.orchestra.run.smithi026.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 661 kB/s | 16 kB 00:00 2024-09-16T16:37:40.469 INFO:teuthology.orchestra.run.smithi026.stdout:(31/37): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-09-16T16:37:40.494 INFO:teuthology.orchestra.run.smithi026.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-16T16:37:40.528 INFO:teuthology.orchestra.run.smithi026.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 608 kB/s | 36 kB 00:00 2024-09-16T16:37:40.554 INFO:teuthology.orchestra.run.smithi026.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-09-16T16:37:40.580 INFO:teuthology.orchestra.run.smithi026.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 804 kB/s | 20 kB 00:00 2024-09-16T16:37:40.604 INFO:teuthology.orchestra.run.smithi026.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.5 MB/s | 427 kB 00:00 2024-09-16T16:37:41.563 INFO:teuthology.orchestra.run.smithi026.stdout:(37/37): ceph-osd-19.3.0-4879.g81fedcb3.el9.x86 29 MB/s | 70 MB 00:02 2024-09-16T16:37:41.568 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:37:41.569 INFO:teuthology.orchestra.run.smithi026.stdout:Total 28 MB/s | 84 MB 00:02 2024-09-16T16:37:41.722 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-16T16:37:41.769 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-16T16:37:41.769 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-16T16:37:42.610 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-16T16:37:42.610 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-16T16:37:44.305 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-16T16:37:44.392 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-16T16:37:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-16T16:37:44.451 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-16T16:37:44.527 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-16T16:37:44.568 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-16T16:37:44.600 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-16T16:37:44.699 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-16T16:37:44.773 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-16T16:37:44.829 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-16T16:37:44.863 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-16T16:37:44.892 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-16T16:37:44.944 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-16T16:37:44.979 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-16T16:37:45.018 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-16T16:37:45.061 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-16T16:37:45.096 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-16T16:37:45.145 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-16T16:37:45.184 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-16T16:37:45.270 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-16T16:37:45.423 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-16T16:37:45.506 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-16T16:37:45.563 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-16T16:37:45.600 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-16T16:37:45.659 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-16T16:37:45.731 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-16T16:37:45.932 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-16T16:37:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-16T16:37:46.070 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-16T16:37:46.167 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-16T16:37:46.218 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-16T16:37:46.361 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-16T16:37:46.558 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4879.g81fedcb3.el9. 32/37 2024-09-16T16:37:46.597 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-2:19.3.0-4879.g81fedcb3.el9.x86_64 33/37 2024-09-16T16:37:46.624 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4879.g81fedcb3.el9.x86_64 33/37 2024-09-16T16:37:46.624 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T16:37:46.624 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T16:37:46.624 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T16:37:46.624 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T16:37:46.625 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:37:57.879 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-osd-2:19.3.0-4879.g81fedcb3.el9.x86_64 34/37 2024-09-16T16:37:57.908 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-osd-2:19.3.0-4879.g81fedcb3.el9.x86_64 34/37 2024-09-16T16:37:57.908 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T16:37:57.908 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T16:37:57.908 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T16:37:57.908 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T16:37:57.909 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:37:58.497 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mon-2:19.3.0-4879.g81fedcb3.el9.x86_64 35/37 2024-09-16T16:37:58.523 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mon-2:19.3.0-4879.g81fedcb3.el9.x86_64 35/37 2024-09-16T16:37:58.523 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T16:37:58.523 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T16:37:58.523 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T16:37:58.523 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T16:37:58.523 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:37:58.772 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mds-2:19.3.0-4879.g81fedcb3.el9.x86_64 36/37 2024-09-16T16:37:58.797 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mds-2:19.3.0-4879.g81fedcb3.el9.x86_64 36/37 2024-09-16T16:37:58.797 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T16:37:58.797 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T16:37:58.797 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T16:37:58.798 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T16:37:58.798 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:37:58.865 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-2:19.3.0-4879.g81fedcb3.el9.x86_64 37/37 2024-09-16T16:38:05.379 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-2:19.3.0-4879.g81fedcb3.el9.x86_64 37/37 2024-09-16T16:38:05.379 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/37 2024-09-16T16:38:05.379 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mds-2:19.3.0-4879.g81fedcb3.el9.x86_64 2/37 2024-09-16T16:38:05.379 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-2:19.3.0-4879.g81fedcb3.el9.x86_64 3/37 2024-09-16T16:38:05.379 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mon-2:19.3.0-4879.g81fedcb3.el9.x86_64 4/37 2024-09-16T16:38:05.379 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-osd-2:19.3.0-4879.g81fedcb3.el9.x86_64 5/37 2024-09-16T16:38:05.379 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4879.g81fedcb3.el9. 6/37 2024-09-16T16:38:05.379 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-16T16:38:05.379 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-16T16:38:05.379 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-16T16:38:05.380 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-16T16:38:05.380 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-16T16:38:05.380 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-16T16:38:05.380 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-16T16:38:05.380 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-16T16:38:05.380 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-16T16:38:05.380 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-16T16:38:05.380 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-16T16:38:05.380 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-16T16:38:05.380 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-16T16:38:05.380 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-16T16:38:05.380 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-16T16:38:05.381 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-16T16:38:05.381 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-16T16:38:05.381 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-16T16:38:05.381 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-16T16:38:05.382 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-16T16:38:05.382 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-16T16:38:05.382 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-16T16:38:05.382 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-16T16:38:05.382 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-16T16:38:05.382 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-16T16:38:05.383 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-16T16:38:05.383 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-16T16:38:05.383 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-16T16:38:05.383 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-16T16:38:05.383 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-16T16:38:06.461 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-16T16:38:06.461 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:06.461 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-16T16:38:06.461 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:38:06.461 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mds-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:38:06.461 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:38:06.461 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-modules-core-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T16:38:06.461 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mon-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:38:06.461 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-osd-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:38:06.461 INFO:teuthology.orchestra.run.smithi026.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T16:38:06.461 INFO:teuthology.orchestra.run.smithi026.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T16:38:06.462 INFO:teuthology.orchestra.run.smithi026.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T16:38:06.462 INFO:teuthology.orchestra.run.smithi026.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T16:38:06.462 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T16:38:06.462 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T16:38:06.462 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T16:38:06.462 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T16:38:06.462 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T16:38:06.462 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T16:38:06.462 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T16:38:06.462 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T16:38:06.462 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T16:38:06.462 INFO:teuthology.orchestra.run.smithi026.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T16:38:06.463 INFO:teuthology.orchestra.run.smithi026.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T16:38:06.463 INFO:teuthology.orchestra.run.smithi026.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T16:38:06.463 INFO:teuthology.orchestra.run.smithi026.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T16:38:06.463 INFO:teuthology.orchestra.run.smithi026.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T16:38:06.463 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T16:38:06.463 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T16:38:06.463 INFO:teuthology.orchestra.run.smithi026.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T16:38:06.463 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T16:38:06.463 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T16:38:06.463 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T16:38:06.463 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T16:38:06.463 INFO:teuthology.orchestra.run.smithi026.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T16:38:06.463 INFO:teuthology.orchestra.run.smithi026.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T16:38:06.464 INFO:teuthology.orchestra.run.smithi026.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T16:38:06.464 INFO:teuthology.orchestra.run.smithi026.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T16:38:06.464 INFO:teuthology.orchestra.run.smithi026.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T16:38:06.464 INFO:teuthology.orchestra.run.smithi026.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T16:38:06.464 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:06.464 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-16T16:38:06.728 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-base 2024-09-16T16:38:07.338 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:25 ago on Mon 16 Sep 2024 04:36:42 PM UTC. 2024-09-16T16:38:07.431 INFO:teuthology.orchestra.run.smithi026.stdout:Package ceph-base-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T16:38:07.476 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-16T16:38:07.477 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-09-16T16:38:07.477 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-16T16:38:07.554 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install cephadm 2024-09-16T16:38:08.139 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:26 ago on Mon 16 Sep 2024 04:36:42 PM UTC. 2024-09-16T16:38:08.277 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-16T16:38:08.278 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:38:08.278 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-09-16T16:38:08.278 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:38:08.278 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-16T16:38:08.278 INFO:teuthology.orchestra.run.smithi026.stdout: cephadm noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 785 k 2024-09-16T16:38:08.279 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:08.279 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-16T16:38:08.279 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:38:08.279 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-09-16T16:38:08.279 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:08.279 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 785 k 2024-09-16T16:38:08.279 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 791 k 2024-09-16T16:38:08.279 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-16T16:38:08.555 INFO:teuthology.orchestra.run.smithi026.stdout:cephadm-19.3.0-4879.g81fedcb3.el9.noarch.rpm 2.8 MB/s | 785 kB 00:00 2024-09-16T16:38:08.556 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:38:08.556 INFO:teuthology.orchestra.run.smithi026.stdout:Total 2.8 MB/s | 785 kB 00:00 2024-09-16T16:38:08.556 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-16T16:38:08.561 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-16T16:38:08.561 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-16T16:38:08.568 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-16T16:38:08.568 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-16T16:38:08.647 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-16T16:38:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 1/1 2024-09-16T16:38:09.169 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 1/1 2024-09-16T16:38:09.703 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 1/1 2024-09-16T16:38:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 1/1 2024-09-16T16:38:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:09.966 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-16T16:38:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T16:38:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:09.966 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-16T16:38:10.187 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T16:38:10.776 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:28 ago on Mon 16 Sep 2024 04:36:42 PM UTC. 2024-09-16T16:38:10.915 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-16T16:38:10.915 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:38:10.915 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-09-16T16:38:10.915 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:38:10.915 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-16T16:38:10.916 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 158 k 2024-09-16T16:38:10.916 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:10.916 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-16T16:38:10.916 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:38:10.916 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-09-16T16:38:10.916 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:10.916 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 158 k 2024-09-16T16:38:10.917 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 644 k 2024-09-16T16:38:10.917 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-16T16:38:11.113 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-immutable-object-cache-19.3.0-4879.g81fedc 806 kB/s | 158 kB 00:00 2024-09-16T16:38:11.113 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:38:11.113 INFO:teuthology.orchestra.run.smithi026.stdout:Total 800 kB/s | 158 kB 00:00 2024-09-16T16:38:11.114 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-16T16:38:11.125 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-16T16:38:11.125 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-16T16:38:11.185 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-16T16:38:11.185 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-16T16:38:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-16T16:38:11.443 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4879.g81fedcb3. 1/1 2024-09-16T16:38:11.466 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4879.g81fedcb3. 1/1 2024-09-16T16:38:11.466 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T16:38:11.466 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T16:38:11.466 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:12.518 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4879.g81fedcb3. 1/1 2024-09-16T16:38:12.518 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:12.518 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-16T16:38:12.518 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-immutable-object-cache-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:38:12.518 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:12.519 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-16T16:38:12.748 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr 2024-09-16T16:38:13.344 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:31 ago on Mon 16 Sep 2024 04:36:42 PM UTC. 2024-09-16T16:38:13.436 INFO:teuthology.orchestra.run.smithi026.stdout:Package ceph-mgr-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T16:38:13.482 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-16T16:38:13.483 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-09-16T16:38:13.483 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-16T16:38:13.551 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T16:38:14.134 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:32 ago on Mon 16 Sep 2024 04:36:42 PM UTC. 2024-09-16T16:38:14.275 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-16T16:38:14.275 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:38:14.275 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-09-16T16:38:14.276 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:38:14.276 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-16T16:38:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 14 M 2024-09-16T16:38:14.276 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-09-16T16:38:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T16:38:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 34 k 2024-09-16T16:38:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 16 k 2024-09-16T16:38:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T16:38:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T16:38:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T16:38:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T16:38:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T16:38:14.277 INFO:teuthology.orchestra.run.smithi026.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T16:38:14.277 INFO:teuthology.orchestra.run.smithi026.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T16:38:14.277 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:14.277 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-16T16:38:14.277 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:38:14.277 INFO:teuthology.orchestra.run.smithi026.stdout:Install 11 Packages 2024-09-16T16:38:14.277 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:14.277 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 18 M 2024-09-16T16:38:14.277 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 155 M 2024-09-16T16:38:14.277 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-16T16:38:14.842 INFO:teuthology.orchestra.run.smithi026.stdout:(1/11): ceph-grafana-dashboards-19.3.0-4879.g81 124 kB/s | 34 kB 00:00 2024-09-16T16:38:14.867 INFO:teuthology.orchestra.run.smithi026.stdout:(2/11): ceph-prometheus-alerts-19.3.0-4879.g81f 55 kB/s | 16 kB 00:00 2024-09-16T16:38:15.041 INFO:teuthology.orchestra.run.smithi026.stdout:(3/11): protobuf-compiler-3.14.0-13.el9.x86_64. 4.9 MB/s | 863 kB 00:00 2024-09-16T16:38:15.101 INFO:teuthology.orchestra.run.smithi026.stdout:(4/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.0 MB/s | 269 kB 00:00 2024-09-16T16:38:15.143 INFO:teuthology.orchestra.run.smithi026.stdout:(5/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 5.3 MB/s | 548 kB 00:00 2024-09-16T16:38:15.168 INFO:teuthology.orchestra.run.smithi026.stdout:(6/11): grpc-data-1.46.7-10.el9.noarch.rpm 290 kB/s | 19 kB 00:00 2024-09-16T16:38:15.203 INFO:teuthology.orchestra.run.smithi026.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 4.2 MB/s | 144 kB 00:00 2024-09-16T16:38:15.228 INFO:teuthology.orchestra.run.smithi026.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-16T16:38:15.269 INFO:teuthology.orchestra.run.smithi026.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 4.4 MB/s | 188 kB 00:00 2024-09-16T16:38:15.327 INFO:teuthology.orchestra.run.smithi026.stdout:(10/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 11 MB/s | 2.0 MB 00:00 2024-09-16T16:38:15.586 INFO:teuthology.orchestra.run.smithi026.stdout:(11/11): ceph-mgr-dashboard-19.3.0-4879.g81fedc 14 MB/s | 14 MB 00:01 2024-09-16T16:38:15.590 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:38:15.590 INFO:teuthology.orchestra.run.smithi026.stdout:Total 14 MB/s | 18 MB 00:01 2024-09-16T16:38:15.622 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-16T16:38:15.643 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-16T16:38:15.643 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-16T16:38:15.927 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-16T16:38:15.928 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-16T16:38:16.497 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-16T16:38:16.578 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-09-16T16:38:16.615 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-09-16T16:38:16.720 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-09-16T16:38:16.855 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-09-16T16:38:16.902 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-09-16T16:38:16.990 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-09-16T16:38:17.038 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-09-16T16:38:17.073 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-09-16T16:38:17.117 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4879.g81fedcb3.el9 9/11 2024-09-16T16:38:20.515 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4879.g81fedcb3.el 10/11 2024-09-16T16:38:20.576 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4879.g81fedcb3.el9.noa 11/11 2024-09-16T16:38:22.475 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4879.g81fedcb3.el9.noa 11/11 2024-09-16T16:38:22.475 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4879.g81fedcb3.el 1/11 2024-09-16T16:38:22.475 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4879.g81fedcb3.el9.noa 2/11 2024-09-16T16:38:22.475 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4879.g81fedcb3.el9 3/11 2024-09-16T16:38:22.475 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-09-16T16:38:22.475 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-09-16T16:38:22.475 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-09-16T16:38:22.475 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-09-16T16:38:22.475 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-09-16T16:38:22.476 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-09-16T16:38:22.476 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-09-16T16:38:22.799 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-09-16T16:38:22.799 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:22.800 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-16T16:38:22.800 INFO:teuthology.orchestra.run.smithi026.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T16:38:22.800 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-grafana-dashboards-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T16:38:22.800 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-dashboard-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T16:38:22.800 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-prometheus-alerts-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T16:38:22.800 INFO:teuthology.orchestra.run.smithi026.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T16:38:22.800 INFO:teuthology.orchestra.run.smithi026.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T16:38:22.800 INFO:teuthology.orchestra.run.smithi026.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T16:38:22.800 INFO:teuthology.orchestra.run.smithi026.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T16:38:22.801 INFO:teuthology.orchestra.run.smithi026.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T16:38:22.801 INFO:teuthology.orchestra.run.smithi026.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T16:38:22.801 INFO:teuthology.orchestra.run.smithi026.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T16:38:22.801 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:22.801 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-16T16:38:23.058 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T16:38:23.646 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:41 ago on Mon 16 Sep 2024 04:36:42 PM UTC. 2024-09-16T16:38:23.785 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-16T16:38:23.786 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================= 2024-09-16T16:38:23.786 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-09-16T16:38:23.786 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================= 2024-09-16T16:38:23.786 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-16T16:38:23.786 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 7.4 M 2024-09-16T16:38:23.786 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-09-16T16:38:23.786 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T16:38:23.786 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T16:38:23.786 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T16:38:23.787 INFO:teuthology.orchestra.run.smithi026.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T16:38:23.787 INFO:teuthology.orchestra.run.smithi026.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T16:38:23.787 INFO:teuthology.orchestra.run.smithi026.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T16:38:23.787 INFO:teuthology.orchestra.run.smithi026.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T16:38:23.787 INFO:teuthology.orchestra.run.smithi026.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T16:38:23.787 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T16:38:23.787 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T16:38:23.787 INFO:teuthology.orchestra.run.smithi026.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T16:38:23.787 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:23.787 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-16T16:38:23.787 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================= 2024-09-16T16:38:23.787 INFO:teuthology.orchestra.run.smithi026.stdout:Install 12 Packages 2024-09-16T16:38:23.787 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:23.788 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 38 M 2024-09-16T16:38:23.788 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 212 M 2024-09-16T16:38:23.788 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-16T16:38:23.975 INFO:teuthology.orchestra.run.smithi026.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 191 kB 00:00 2024-09-16T16:38:24.042 INFO:teuthology.orchestra.run.smithi026.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.8 MB/s | 801 kB 00:00 2024-09-16T16:38:24.067 INFO:teuthology.orchestra.run.smithi026.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 357 kB/s | 33 kB 00:00 2024-09-16T16:38:24.109 INFO:teuthology.orchestra.run.smithi026.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 438 kB/s | 18 kB 00:00 2024-09-16T16:38:24.168 INFO:teuthology.orchestra.run.smithi026.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 660 kB/s | 38 kB 00:00 2024-09-16T16:38:24.302 INFO:teuthology.orchestra.run.smithi026.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-48 16 MB/s | 7.4 MB 00:00 2024-09-16T16:38:24.352 INFO:teuthology.orchestra.run.smithi026.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.7 MB/s | 3.0 MB 00:00 2024-09-16T16:38:24.489 INFO:teuthology.orchestra.run.smithi026.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 16 MB/s | 4.9 MB 00:00 2024-09-16T16:38:24.521 INFO:teuthology.orchestra.run.smithi026.stdout:(9/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-09-16T16:38:24.544 INFO:teuthology.orchestra.run.smithi026.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 4.2 MB/s | 250 kB 00:00 2024-09-16T16:38:24.628 INFO:teuthology.orchestra.run.smithi026.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 5.1 MB 00:00 2024-09-16T16:38:24.870 INFO:teuthology.orchestra.run.smithi026.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-09-16T16:38:24.871 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:38:24.871 INFO:teuthology.orchestra.run.smithi026.stdout:Total 35 MB/s | 38 MB 00:01 2024-09-16T16:38:25.234 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-16T16:38:25.261 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-16T16:38:25.261 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-16T16:38:25.726 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-16T16:38:25.727 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-16T16:38:26.639 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-16T16:38:26.718 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T16:38:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T16:38:26.872 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T16:38:27.018 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T16:38:27.205 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T16:38:27.270 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T16:38:27.357 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T16:38:28.050 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T16:38:28.172 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T16:38:29.473 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T16:38:30.559 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T16:38:30.583 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4879.g81fed 12/12 2024-09-16T16:38:33.100 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4879.g81fed 12/12 2024-09-16T16:38:33.100 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4879.g81fed 1/12 2024-09-16T16:38:33.100 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T16:38:33.100 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T16:38:33.100 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T16:38:33.101 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T16:38:33.101 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T16:38:33.101 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T16:38:33.101 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T16:38:33.101 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T16:38:33.101 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T16:38:33.101 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T16:38:33.547 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T16:38:33.547 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:33.547 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-16T16:38:33.547 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T16:38:33.547 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T16:38:33.547 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T16:38:33.547 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T16:38:33.548 INFO:teuthology.orchestra.run.smithi026.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T16:38:33.548 INFO:teuthology.orchestra.run.smithi026.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T16:38:33.548 INFO:teuthology.orchestra.run.smithi026.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T16:38:33.548 INFO:teuthology.orchestra.run.smithi026.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T16:38:33.548 INFO:teuthology.orchestra.run.smithi026.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T16:38:33.548 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T16:38:33.548 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T16:38:33.548 INFO:teuthology.orchestra.run.smithi026.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T16:38:33.548 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:33.548 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-16T16:38:33.946 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-rook 2024-09-16T16:38:34.534 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:52 ago on Mon 16 Sep 2024 04:36:42 PM UTC. 2024-09-16T16:38:34.673 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-16T16:38:34.674 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:38:34.675 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-09-16T16:38:34.675 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:38:34.675 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-16T16:38:34.675 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-rook noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 49 k 2024-09-16T16:38:34.675 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-09-16T16:38:34.675 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T16:38:34.675 INFO:teuthology.orchestra.run.smithi026.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T16:38:34.675 INFO:teuthology.orchestra.run.smithi026.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T16:38:34.675 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T16:38:34.675 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T16:38:34.675 INFO:teuthology.orchestra.run.smithi026.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T16:38:34.675 INFO:teuthology.orchestra.run.smithi026.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T16:38:34.676 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T16:38:34.676 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T16:38:34.676 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T16:38:34.676 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T16:38:34.676 INFO:teuthology.orchestra.run.smithi026.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T16:38:34.676 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:34.676 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-16T16:38:34.676 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:38:34.676 INFO:teuthology.orchestra.run.smithi026.stdout:Install 13 Packages 2024-09-16T16:38:34.676 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:34.677 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 2.2 M 2024-09-16T16:38:34.677 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 26 M 2024-09-16T16:38:34.677 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-16T16:38:34.966 INFO:teuthology.orchestra.run.smithi026.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 200 kB/s | 26 kB 00:00 2024-09-16T16:38:34.992 INFO:teuthology.orchestra.run.smithi026.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 125 kB/s | 19 kB 00:00 2024-09-16T16:38:35.017 INFO:teuthology.orchestra.run.smithi026.stdout:(3/13): ceph-mgr-rook-19.3.0-4879.g81fedcb3.el9 276 kB/s | 49 kB 00:00 2024-09-16T16:38:35.086 INFO:teuthology.orchestra.run.smithi026.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 222 kB 00:00 2024-09-16T16:38:35.109 INFO:teuthology.orchestra.run.smithi026.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-16T16:38:35.160 INFO:teuthology.orchestra.run.smithi026.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 644 kB/s | 32 kB 00:00 2024-09-16T16:38:35.189 INFO:teuthology.orchestra.run.smithi026.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 532 kB/s | 54 kB 00:00 2024-09-16T16:38:35.211 INFO:teuthology.orchestra.run.smithi026.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 281 kB/s | 14 kB 00:00 2024-09-16T16:38:35.261 INFO:teuthology.orchestra.run.smithi026.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.1 MB/s | 279 kB 00:00 2024-09-16T16:38:35.311 INFO:teuthology.orchestra.run.smithi026.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 1.7 MB/s | 220 kB 00:00 2024-09-16T16:38:35.336 INFO:teuthology.orchestra.run.smithi026.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 808 kB/s | 59 kB 00:00 2024-09-16T16:38:35.361 INFO:teuthology.orchestra.run.smithi026.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-16T16:38:35.395 INFO:teuthology.orchestra.run.smithi026.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.8 MB/s | 1.0 MB 00:00 2024-09-16T16:38:35.397 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:38:35.398 INFO:teuthology.orchestra.run.smithi026.stdout:Total 3.1 MB/s | 2.2 MB 00:00 2024-09-16T16:38:35.463 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-16T16:38:35.479 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-16T16:38:35.480 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-16T16:38:35.576 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-16T16:38:35.578 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-16T16:38:36.095 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-16T16:38:36.266 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T16:38:36.323 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T16:38:36.387 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T16:38:36.422 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T16:38:36.473 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T16:38:36.588 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T16:38:36.697 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T16:38:36.758 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T16:38:37.529 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T16:38:37.593 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T16:38:37.621 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T16:38:37.671 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T16:38:37.699 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-rook-2:19.3.0-4879.g81fedcb3.el9.noarch 13/13 2024-09-16T16:38:38.471 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4879.g81fedcb3.el9.noarch 13/13 2024-09-16T16:38:38.472 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4879.g81fedcb3.el9.noarch 1/13 2024-09-16T16:38:38.472 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T16:38:38.472 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T16:38:38.472 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T16:38:38.472 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T16:38:38.472 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T16:38:38.472 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T16:38:38.472 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T16:38:38.472 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T16:38:38.472 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T16:38:38.472 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T16:38:38.472 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T16:38:38.830 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T16:38:38.830 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:38.830 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-16T16:38:38.830 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-rook-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T16:38:38.830 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T16:38:38.830 INFO:teuthology.orchestra.run.smithi026.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T16:38:38.831 INFO:teuthology.orchestra.run.smithi026.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T16:38:38.831 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T16:38:38.831 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T16:38:38.831 INFO:teuthology.orchestra.run.smithi026.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T16:38:38.831 INFO:teuthology.orchestra.run.smithi026.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T16:38:38.831 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T16:38:38.831 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T16:38:38.831 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T16:38:38.831 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T16:38:38.831 INFO:teuthology.orchestra.run.smithi026.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T16:38:38.831 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:38.831 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-16T16:38:39.074 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T16:38:39.687 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:57 ago on Mon 16 Sep 2024 04:36:42 PM UTC. 2024-09-16T16:38:39.827 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-16T16:38:39.827 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:38:39.827 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-09-16T16:38:39.828 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:38:39.828 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-16T16:38:39.828 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 154 k 2024-09-16T16:38:39.828 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-09-16T16:38:39.828 INFO:teuthology.orchestra.run.smithi026.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T16:38:39.828 INFO:teuthology.orchestra.run.smithi026.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T16:38:39.828 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T16:38:39.828 INFO:teuthology.orchestra.run.smithi026.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T16:38:39.828 INFO:teuthology.orchestra.run.smithi026.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T16:38:39.828 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:39.828 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-16T16:38:39.828 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:38:39.828 INFO:teuthology.orchestra.run.smithi026.stdout:Install 6 Packages 2024-09-16T16:38:39.829 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:39.829 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 7.0 M 2024-09-16T16:38:39.830 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 33 M 2024-09-16T16:38:39.830 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-16T16:38:40.093 INFO:teuthology.orchestra.run.smithi026.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-09-16T16:38:40.115 INFO:teuthology.orchestra.run.smithi026.stdout:(2/6): ceph-mgr-cephadm-19.3.0-4879.g81fedcb3.e 683 kB/s | 154 kB 00:00 2024-09-16T16:38:40.173 INFO:teuthology.orchestra.run.smithi026.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 990 kB/s | 58 kB 00:00 2024-09-16T16:38:40.207 INFO:teuthology.orchestra.run.smithi026.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.6 MB/s | 548 kB 00:00 2024-09-16T16:38:40.232 INFO:teuthology.orchestra.run.smithi026.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.2 MB/s | 74 kB 00:00 2024-09-16T16:38:41.165 INFO:teuthology.orchestra.run.smithi026.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.7 MB/s | 6.0 MB 00:01 2024-09-16T16:38:41.168 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:38:41.168 INFO:teuthology.orchestra.run.smithi026.stdout:Total 5.2 MB/s | 7.0 MB 00:01 2024-09-16T16:38:41.241 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-16T16:38:41.253 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-16T16:38:41.253 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-16T16:38:41.359 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-16T16:38:41.359 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-16T16:38:41.685 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-16T16:38:41.867 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T16:38:41.927 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T16:38:42.479 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T16:38:42.574 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T16:38:42.657 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T16:38:42.688 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 6/6 2024-09-16T16:38:43.469 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 6/6 2024-09-16T16:38:43.469 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 1/6 2024-09-16T16:38:43.469 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T16:38:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T16:38:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T16:38:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T16:38:43.756 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T16:38:43.756 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:43.757 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-16T16:38:43.757 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T16:38:43.757 INFO:teuthology.orchestra.run.smithi026.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T16:38:43.757 INFO:teuthology.orchestra.run.smithi026.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T16:38:43.757 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T16:38:43.757 INFO:teuthology.orchestra.run.smithi026.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T16:38:43.757 INFO:teuthology.orchestra.run.smithi026.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T16:38:43.757 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:43.757 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-16T16:38:43.999 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-fuse 2024-09-16T16:38:44.596 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:02:02 ago on Mon 16 Sep 2024 04:36:42 PM UTC. 2024-09-16T16:38:44.735 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-16T16:38:44.736 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:38:44.736 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-09-16T16:38:44.736 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:38:44.736 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-16T16:38:44.736 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-fuse x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 884 k 2024-09-16T16:38:44.736 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-09-16T16:38:44.736 INFO:teuthology.orchestra.run.smithi026.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T16:38:44.736 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:44.736 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-16T16:38:44.736 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:38:44.737 INFO:teuthology.orchestra.run.smithi026.stdout:Install 2 Packages 2024-09-16T16:38:44.737 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:44.737 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 964 k 2024-09-16T16:38:44.737 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 4.1 M 2024-09-16T16:38:44.737 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-16T16:38:45.142 INFO:teuthology.orchestra.run.smithi026.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 539 kB/s | 80 kB 00:00 2024-09-16T16:38:45.267 INFO:teuthology.orchestra.run.smithi026.stdout:(2/2): ceph-fuse-19.3.0-4879.g81fedcb3.el9.x86_ 3.2 MB/s | 884 kB 00:00 2024-09-16T16:38:45.268 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:38:45.268 INFO:teuthology.orchestra.run.smithi026.stdout:Total 1.8 MB/s | 964 kB 00:00 2024-09-16T16:38:45.281 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-16T16:38:45.308 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-16T16:38:45.308 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-16T16:38:45.397 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-16T16:38:45.397 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-16T16:38:45.593 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-16T16:38:45.754 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T16:38:45.829 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 2/2 2024-09-16T16:38:46.524 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 2/2 2024-09-16T16:38:46.524 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/2 2024-09-16T16:38:46.812 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T16:38:46.812 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:46.812 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-16T16:38:46.812 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T16:38:46.812 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:46.812 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-16T16:38:47.090 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-volume 2024-09-16T16:38:47.687 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:02:05 ago on Mon 16 Sep 2024 04:36:42 PM UTC. 2024-09-16T16:38:47.825 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-16T16:38:47.826 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:38:47.826 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-09-16T16:38:47.826 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:38:47.826 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-16T16:38:47.826 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-volume noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 285 k 2024-09-16T16:38:47.826 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:47.826 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-16T16:38:47.826 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:38:47.826 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-09-16T16:38:47.826 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:47.826 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 285 k 2024-09-16T16:38:47.827 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 1.3 M 2024-09-16T16:38:47.827 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-16T16:38:48.056 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-volume-19.3.0-4879.g81fedcb3.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-16T16:38:48.057 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:38:48.057 INFO:teuthology.orchestra.run.smithi026.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-16T16:38:48.057 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-16T16:38:48.066 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-16T16:38:48.066 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-16T16:38:48.097 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-16T16:38:48.097 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-16T16:38:48.331 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-16T16:38:48.400 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-volume-2:19.3.0-4879.g81fedcb3.el9.noarch 1/1 2024-09-16T16:38:48.417 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-volume-2:19.3.0-4879.g81fedcb3.el9.noarch 1/1 2024-09-16T16:38:48.417 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T16:38:48.417 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T16:38:48.417 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:49.435 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-volume-2:19.3.0-4879.g81fedcb3.el9.noarch 1/1 2024-09-16T16:38:49.435 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:49.435 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-16T16:38:49.435 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-volume-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T16:38:49.435 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:49.436 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-16T16:38:49.745 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librados-devel 2024-09-16T16:38:50.342 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:02:08 ago on Mon 16 Sep 2024 04:36:42 PM UTC. 2024-09-16T16:38:50.484 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-16T16:38:50.485 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:38:50.485 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-09-16T16:38:50.485 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:38:50.485 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-16T16:38:50.485 INFO:teuthology.orchestra.run.smithi026.stdout: librados-devel x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 125 k 2024-09-16T16:38:50.486 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:50.486 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-16T16:38:50.486 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:38:50.486 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-09-16T16:38:50.486 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:50.486 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 125 k 2024-09-16T16:38:50.486 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 445 k 2024-09-16T16:38:50.486 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-16T16:38:50.669 INFO:teuthology.orchestra.run.smithi026.stdout:librados-devel-19.3.0-4879.g81fedcb3.el9.x86_64 681 kB/s | 125 kB 00:00 2024-09-16T16:38:50.670 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:38:50.670 INFO:teuthology.orchestra.run.smithi026.stdout:Total 675 kB/s | 125 kB 00:00 2024-09-16T16:38:50.670 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-16T16:38:50.677 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-16T16:38:50.677 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-16T16:38:50.744 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-16T16:38:50.744 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-16T16:38:50.908 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-16T16:38:51.012 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librados-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:38:51.456 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:38:51.720 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:38:51.720 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:51.720 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-16T16:38:51.720 INFO:teuthology.orchestra.run.smithi026.stdout: librados-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:38:51.721 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:51.721 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-16T16:38:51.991 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install libcephfs2 2024-09-16T16:38:52.586 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:02:10 ago on Mon 16 Sep 2024 04:36:42 PM UTC. 2024-09-16T16:38:52.679 INFO:teuthology.orchestra.run.smithi026.stdout:Package libcephfs2-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T16:38:52.725 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-16T16:38:52.726 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-09-16T16:38:52.726 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-16T16:38:52.804 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install libcephfs-devel 2024-09-16T16:38:53.409 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:02:11 ago on Mon 16 Sep 2024 04:36:42 PM UTC. 2024-09-16T16:38:53.543 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-16T16:38:53.543 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:38:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-09-16T16:38:53.544 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:38:53.544 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-16T16:38:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs-devel x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 32 k 2024-09-16T16:38:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:53.544 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-16T16:38:53.544 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:38:53.544 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-09-16T16:38:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:53.544 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 32 k 2024-09-16T16:38:53.544 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 142 k 2024-09-16T16:38:53.544 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-16T16:38:53.705 INFO:teuthology.orchestra.run.smithi026.stdout:libcephfs-devel-19.3.0-4879.g81fedcb3.el9.x86_6 197 kB/s | 32 kB 00:00 2024-09-16T16:38:53.706 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:38:53.706 INFO:teuthology.orchestra.run.smithi026.stdout:Total 196 kB/s | 32 kB 00:00 2024-09-16T16:38:53.706 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-16T16:38:53.710 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-16T16:38:53.710 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-16T16:38:53.727 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-16T16:38:53.727 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-16T16:38:53.809 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-16T16:38:53.940 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephfs-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:38:54.313 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:38:54.543 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephfs-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:38:54.543 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:54.543 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-16T16:38:54.543 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:38:54.543 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:38:54.543 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-16T16:38:54.731 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librados2 2024-09-16T16:38:55.329 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:02:13 ago on Mon 16 Sep 2024 04:36:42 PM UTC. 2024-09-16T16:38:55.417 INFO:teuthology.orchestra.run.smithi026.stdout:Package librados2-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T16:38:55.463 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-16T16:38:55.464 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-09-16T16:38:55.464 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-16T16:38:55.539 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librbd1 2024-09-16T16:38:56.132 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:02:14 ago on Mon 16 Sep 2024 04:36:42 PM UTC. 2024-09-16T16:38:56.226 INFO:teuthology.orchestra.run.smithi026.stdout:Package librbd1-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T16:38:56.272 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-16T16:38:56.273 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-09-16T16:38:56.274 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-16T16:38:56.349 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rados 2024-09-16T16:38:56.949 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:02:14 ago on Mon 16 Sep 2024 04:36:42 PM UTC. 2024-09-16T16:38:57.042 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-rados-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T16:38:57.089 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-16T16:38:57.089 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-09-16T16:38:57.090 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-16T16:38:57.168 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rgw 2024-09-16T16:38:57.759 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:02:15 ago on Mon 16 Sep 2024 04:36:42 PM UTC. 2024-09-16T16:38:57.851 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-rgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T16:38:57.897 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-16T16:38:57.898 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-09-16T16:38:57.898 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-16T16:38:57.970 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-cephfs 2024-09-16T16:38:58.566 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:02:16 ago on Mon 16 Sep 2024 04:36:42 PM UTC. 2024-09-16T16:38:58.658 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-cephfs-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T16:38:58.704 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-16T16:38:58.705 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-09-16T16:38:58.706 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-16T16:38:58.782 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rbd 2024-09-16T16:38:59.381 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:02:17 ago on Mon 16 Sep 2024 04:36:42 PM UTC. 2024-09-16T16:38:59.478 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-rbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T16:38:59.528 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-16T16:38:59.529 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-09-16T16:38:59.529 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-16T16:38:59.606 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-fuse 2024-09-16T16:39:00.197 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:02:18 ago on Mon 16 Sep 2024 04:36:42 PM UTC. 2024-09-16T16:39:00.338 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-16T16:39:00.339 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:39:00.339 INFO:teuthology.orchestra.run.smithi026.stdout: Package Architecture Version Repository Size 2024-09-16T16:39:00.339 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:39:00.339 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-16T16:39:00.339 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-fuse x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 91 k 2024-09-16T16:39:00.339 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:00.339 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-16T16:39:00.340 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:39:00.340 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-09-16T16:39:00.340 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:00.340 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 91 k 2024-09-16T16:39:00.340 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 321 k 2024-09-16T16:39:00.340 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-16T16:39:00.533 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-fuse-19.3.0-4879.g81fedcb3.el9.x86_64.rpm 471 kB/s | 91 kB 00:00 2024-09-16T16:39:00.534 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:39:00.534 INFO:teuthology.orchestra.run.smithi026.stdout:Total 468 kB/s | 91 kB 00:00 2024-09-16T16:39:00.534 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-16T16:39:00.544 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-16T16:39:00.544 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-16T16:39:00.612 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-16T16:39:00.612 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-16T16:39:00.767 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-16T16:39:00.885 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:39:01.253 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:39:01.517 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:39:01.517 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:01.517 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-16T16:39:01.518 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:39:01.518 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:01.518 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-16T16:39:01.779 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-mirror 2024-09-16T16:39:02.381 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:02:20 ago on Mon 16 Sep 2024 04:36:42 PM UTC. 2024-09-16T16:39:02.525 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-16T16:39:02.526 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:39:02.526 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-09-16T16:39:02.526 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:39:02.526 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-16T16:39:02.526 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-mirror x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 3.5 M 2024-09-16T16:39:02.526 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:02.526 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-16T16:39:02.526 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:39:02.526 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-09-16T16:39:02.526 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:02.526 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 3.5 M 2024-09-16T16:39:02.526 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 30 M 2024-09-16T16:39:02.527 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-16T16:39:02.870 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-mirror-19.3.0-4879.g81fedcb3.el9.x86_64.rpm 10 MB/s | 3.5 MB 00:00 2024-09-16T16:39:02.870 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:39:02.870 INFO:teuthology.orchestra.run.smithi026.stdout:Total 10 MB/s | 3.5 MB 00:00 2024-09-16T16:39:02.871 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-16T16:39:02.884 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-16T16:39:02.884 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-16T16:39:02.972 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-16T16:39:02.972 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-16T16:39:03.597 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-16T16:39:03.663 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-mirror-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:39:03.689 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:39:03.689 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T16:39:03.689 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T16:39:03.689 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T16:39:03.689 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T16:39:03.690 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:04.991 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-mirror-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:39:04.992 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:04.992 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-16T16:39:04.992 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-mirror-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:39:04.992 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:04.992 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-16T16:39:05.244 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-nbd 2024-09-16T16:39:05.843 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:02:23 ago on Mon 16 Sep 2024 04:36:42 PM UTC. 2024-09-16T16:39:05.983 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-16T16:39:05.984 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:39:05.984 INFO:teuthology.orchestra.run.smithi026.stdout: Package Architecture Version Repository Size 2024-09-16T16:39:05.985 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:39:05.985 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-16T16:39:05.985 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-nbd x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 177 k 2024-09-16T16:39:05.985 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:05.985 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-16T16:39:05.985 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-16T16:39:05.985 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-09-16T16:39:05.985 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:05.985 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 177 k 2024-09-16T16:39:05.985 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 685 k 2024-09-16T16:39:05.985 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-16T16:39:06.361 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-nbd-19.3.0-4879.g81fedcb3.el9.x86_64.rpm 477 kB/s | 177 kB 00:00 2024-09-16T16:39:06.361 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-16T16:39:06.361 INFO:teuthology.orchestra.run.smithi026.stdout:Total 475 kB/s | 177 kB 00:00 2024-09-16T16:39:06.361 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-16T16:39:06.375 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-16T16:39:06.375 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-16T16:39:06.442 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-16T16:39:06.442 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-16T16:39:06.633 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-16T16:39:06.732 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-nbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:39:07.184 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:39:07.423 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-nbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T16:39:07.424 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:07.424 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-16T16:39:07.424 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-nbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T16:39:07.424 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:07.424 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-16T16:39:07.748 DEBUG:teuthology.parallel:result is None 2024-09-16T16:39:07.749 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=81fedcb3b484c843d22ffbc8d179cb567b7083b5 2024-09-16T16:39:07.912 DEBUG:teuthology.orchestra.run.smithi026:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T16:39:07.937 INFO:teuthology.orchestra.run.smithi026.stdout:19.3.0-4879.g81fedcb3.el9 2024-09-16T16:39:07.937 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4879.g81fedcb3.el9 2024-09-16T16:39:07.938 INFO:teuthology.task.install:The correct ceph version 19.3.0-4879.g81fedcb3 is installed. 2024-09-16T16:39:07.939 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T16:39:07.940 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T16:39:07.940 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T16:39:08.005 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T16:39:08.005 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T16:39:08.005 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T16:39:08.072 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T16:39:08.140 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T16:39:08.140 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T16:39:08.140 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T16:39:08.206 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T16:39:08.278 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T16:39:08.278 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T16:39:08.278 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T16:39:08.344 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T16:39:08.423 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T16:39:08.685 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T16:39:08.685 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 777 /var/log/ceph 2024-09-16T16:39:08.712 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T16:39:08.713 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T16:39:08.781 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T16:39:08.781 DEBUG:teuthology.orchestra.run.smithi026:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T16:39:08.866 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-16T16:39:08.866 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr stats period': 30}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T16:39:08.866 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2024-09-16_16:23:48-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi/7907402', 'branch': 'wip-matanb-crimson-testing-59543', 'description': 'crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore tasks/rbd_lock_and_fence}', 'email': None, 'first_in_suite': False, 'job_id': '7907402', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2024-09-16_16:23:48-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-matanb-crimson-testing-59543'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr stats period': 30}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '81fedcb3b484c843d22ffbc8d179cb567b7083b5'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '81fedcb3b484c843d22ffbc8d179cb567b7083b5'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '0325fd3c11a225c0fd9eb38fab88aad1370ad779'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 6678, 'sha1': '81fedcb3b484c843d22ffbc8d179cb567b7083b5', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '0325fd3c11a225c0fd9eb38fab88aad1370ad779', 'targets': {'smithi026.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIaeFpgmyXprtzZw6b9E7XAb5is+fCrTfoIGxEOLdv0i2Tkp6JIygOg1w50yJ8ZO+JWzBMBwx0efv8JiJ7SEiA0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr stats period': 30}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '81fedcb3b484c843d22ffbc8d179cb567b7083b5', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['rbd/test_lock_fence.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-16_16:23:48', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T16:39:08.867 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T16:39:08.905 DEBUG:teuthology.orchestra.run.smithi026:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T16:39:08.973 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T16:39:08.973 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T16:39:09.029 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T16:39:09.029 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_1 2024-09-16T16:39:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T16:39:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T16:39:09.088 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-16T16:39:09.088 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T16:39:09.088 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T16:39:09.088 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-09-16 16:39:03.952089009 +0000 2024-09-16T16:39:09.089 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-09-16 16:35:59.187221399 +0000 2024-09-16T16:39:09.089 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-09-16 16:35:59.187221399 +0000 2024-09-16T16:39:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-09-16 16:35:59.187221399 +0000 2024-09-16T16:39:09.089 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T16:39:09.153 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-09-16T16:39:09.153 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-09-16T16:39:09.154 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000100428 s, 5.1 MB/s 2024-09-16T16:39:09.155 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T16:39:09.216 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_2 2024-09-16T16:39:09.276 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T16:39:09.276 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T16:39:09.277 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-16T16:39:09.277 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T16:39:09.277 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T16:39:09.277 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-09-16 16:39:03.953088996 +0000 2024-09-16T16:39:09.277 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-09-16 16:35:59.511217078 +0000 2024-09-16T16:39:09.277 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-09-16 16:35:59.511217078 +0000 2024-09-16T16:39:09.277 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-09-16 16:35:59.511217078 +0000 2024-09-16T16:39:09.277 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T16:39:09.342 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-09-16T16:39:09.342 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-09-16T16:39:09.342 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 8.825e-05 s, 5.8 MB/s 2024-09-16T16:39:09.344 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T16:39:09.410 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_3 2024-09-16T16:39:09.461 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T16:39:09.461 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T16:39:09.461 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-16T16:39:09.461 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T16:39:09.461 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T16:39:09.461 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-09-16 16:39:03.953088996 +0000 2024-09-16T16:39:09.461 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-09-16 16:35:59.851212544 +0000 2024-09-16T16:39:09.462 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-09-16 16:35:59.851212544 +0000 2024-09-16T16:39:09.462 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-09-16 16:35:59.851212544 +0000 2024-09-16T16:39:09.462 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T16:39:09.529 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-09-16T16:39:09.530 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-09-16T16:39:09.530 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.00010183 s, 5.0 MB/s 2024-09-16T16:39:09.530 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T16:39:09.591 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_4 2024-09-16T16:39:09.650 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T16:39:09.650 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T16:39:09.650 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-16T16:39:09.650 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T16:39:09.651 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T16:39:09.651 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-09-16 16:39:03.954088982 +0000 2024-09-16T16:39:09.651 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-09-16 16:36:00.179208171 +0000 2024-09-16T16:39:09.651 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-09-16 16:36:00.179208171 +0000 2024-09-16T16:39:09.651 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-09-16 16:36:00.179208171 +0000 2024-09-16T16:39:09.651 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T16:39:09.714 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-09-16T16:39:09.715 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-09-16T16:39:09.715 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000104966 s, 4.9 MB/s 2024-09-16T16:39:09.716 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T16:39:09.774 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T16:39:09.774 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi026.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-16T16:39:09.775 INFO:tasks.ceph:Generating config... 2024-09-16T16:39:09.776 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-09-16T16:39:09.776 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-16T16:39:09.776 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-16T16:39:09.776 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-16T16:39:09.776 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-09-16T16:39:09.776 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T16:39:09.776 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T16:39:09.776 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T16:39:09.776 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-09-16T16:39:09.777 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-16T16:39:09.777 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T16:39:09.777 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-09-16T16:39:09.777 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-16T16:39:09.777 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T16:39:09.777 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-09-16T16:39:09.777 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-09-16T16:39:09.777 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-09-16T16:39:09.777 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-16T16:39:09.777 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-16T16:39:09.777 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T16:39:09.777 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T16:39:09.777 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-16T16:39:09.777 INFO:tasks.ceph:[osd] err to stderr = True 2024-09-16T16:39:09.778 INFO:tasks.ceph:[osd] log flush on exit = True 2024-09-16T16:39:09.778 INFO:tasks.ceph:[osd] log to file = False 2024-09-16T16:39:09.778 INFO:tasks.ceph:[osd] log to stderr = True 2024-09-16T16:39:09.778 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T16:39:09.778 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T16:39:09.778 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T16:39:09.778 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T16:39:09.778 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T16:39:09.778 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-16T16:39:09.778 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-09-16T16:39:09.778 INFO:tasks.ceph:[global] ms client mode = crc 2024-09-16T16:39:09.778 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-09-16T16:39:09.778 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-09-16T16:39:09.778 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-09-16T16:39:09.779 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-09-16T16:39:09.779 INFO:tasks.ceph:[global] ms service mode = crc 2024-09-16T16:39:09.779 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T16:39:09.779 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T16:39:09.779 INFO:tasks.ceph:[mgr] mgr stats period = 30 2024-09-16T16:39:09.779 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-1 2024-09-16T16:39:09.779 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 2-3 2024-09-16T16:39:09.779 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 4-5 2024-09-16T16:39:09.779 INFO:tasks.ceph:Setting up mon.a... 2024-09-16T16:39:09.779 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-16T16:39:09.884 INFO:teuthology.orchestra.run.smithi026.stdout:creating /etc/ceph/ceph.keyring 2024-09-16T16:39:09.887 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-16T16:39:09.940 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T16:39:09.967 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.26')] 2024-09-16T16:39:09.968 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.26', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'mgr stats period': 30}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}} 2024-09-16T16:39:09.969 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T16:39:09.969 DEBUG:teuthology.orchestra.run.smithi026:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T16:39:10.025 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.26 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-16T16:39:10.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:10.122+0000 7fcdecee7f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:10.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:10.122+0000 7fcdecee7f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:10.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:10.122+0000 7fcdecee7f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:10.123 INFO:teuthology.orchestra.run.smithi026.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-16T16:39:10.123 INFO:teuthology.orchestra.run.smithi026.stdout:monmaptool: generated fsid 3da74dbb-4758-44e9-b920-58aeda43326b 2024-09-16T16:39:10.123 INFO:teuthology.orchestra.run.smithi026.stdout:setting min_mon_release = quincy 2024-09-16T16:39:10.124 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 0 2024-09-16T16:39:10.124 INFO:teuthology.orchestra.run.smithi026.stdout:fsid 3da74dbb-4758-44e9-b920-58aeda43326b 2024-09-16T16:39:10.124 INFO:teuthology.orchestra.run.smithi026.stdout:last_changed 2024-09-16T16:39:10.123638+0000 2024-09-16T16:39:10.124 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-09-16T16:39:10.123638+0000 2024-09-16T16:39:10.124 INFO:teuthology.orchestra.run.smithi026.stdout:min_mon_release 17 (quincy) 2024-09-16T16:39:10.124 INFO:teuthology.orchestra.run.smithi026.stdout:election_strategy: 1 2024-09-16T16:39:10.124 INFO:teuthology.orchestra.run.smithi026.stdout:0: [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] mon.a 2024-09-16T16:39:10.124 INFO:teuthology.orchestra.run.smithi026.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-09-16T16:39:10.125 DEBUG:teuthology.orchestra.run.smithi026:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T16:39:10.142 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 3da74dbb-4758-44e9-b920-58aeda43326b... 2024-09-16T16:39:10.143 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T16:39:10.227 INFO:teuthology.orchestra.run.smithi026.stdout:[global] 2024-09-16T16:39:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: chdir = "" 2024-09-16T16:39:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T16:39:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: auth supported = cephx 2024-09-16T16:39:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: filestore xattr use omap = true 2024-09-16T16:39:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: mon clock drift allowed = 1.000 2024-09-16T16:39:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: osd crush chooseleaf type = 0 2024-09-16T16:39:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: auth debug = true 2024-09-16T16:39:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: ms die on old message = true 2024-09-16T16:39:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: ms die on bug = true 2024-09-16T16:39:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T16:39:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: mon pg warn max object skew = 0 2024-09-16T16:39:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T16:39:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T16:39:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:10.229 INFO:teuthology.orchestra.run.smithi026.stdout: osd pool default size = 2 2024-09-16T16:39:10.229 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:10.229 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd allow primary affinity = true 2024-09-16T16:39:10.229 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd allow pg remap = true 2024-09-16T16:39:10.229 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on legacy crush tunables = false 2024-09-16T16:39:10.229 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on crush straw calc version zero = false 2024-09-16T16:39:10.229 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on no sortbitwise = false 2024-09-16T16:39:10.229 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on osd down out interval zero = false 2024-09-16T16:39:10.229 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on too few osds = false 2024-09-16T16:39:10.229 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T16:39:10.229 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T16:39:10.229 INFO:teuthology.orchestra.run.smithi026.stdout: mon_allow_pool_size_one = true 2024-09-16T16:39:10.230 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:10.230 INFO:teuthology.orchestra.run.smithi026.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T16:39:10.230 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:10.230 INFO:teuthology.orchestra.run.smithi026.stdout: osd default data pool replay window = 5 2024-09-16T16:39:10.230 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:10.230 INFO:teuthology.orchestra.run.smithi026.stdout: mon allow pool delete = true 2024-09-16T16:39:10.230 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:10.230 INFO:teuthology.orchestra.run.smithi026.stdout: mon cluster log file level = debug 2024-09-16T16:39:10.230 INFO:teuthology.orchestra.run.smithi026.stdout: debug asserts on shutdown = true 2024-09-16T16:39:10.230 INFO:teuthology.orchestra.run.smithi026.stdout: mon health detail to clog = false 2024-09-16T16:39:10.230 INFO:teuthology.orchestra.run.smithi026.stdout: mon host = 172.21.15.26 2024-09-16T16:39:10.230 INFO:teuthology.orchestra.run.smithi026.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-09-16T16:39:10.230 INFO:teuthology.orchestra.run.smithi026.stdout: ms client mode = crc 2024-09-16T16:39:10.230 INFO:teuthology.orchestra.run.smithi026.stdout: ms cluster mode = crc 2024-09-16T16:39:10.231 INFO:teuthology.orchestra.run.smithi026.stdout: ms mon client mode = crc 2024-09-16T16:39:10.231 INFO:teuthology.orchestra.run.smithi026.stdout: ms mon cluster mode = crc 2024-09-16T16:39:10.232 INFO:teuthology.orchestra.run.smithi026.stdout: ms mon service mode = crc 2024-09-16T16:39:10.232 INFO:teuthology.orchestra.run.smithi026.stdout: ms service mode = crc 2024-09-16T16:39:10.239 INFO:teuthology.orchestra.run.smithi026.stdout: fsid = 3da74dbb-4758-44e9-b920-58aeda43326b 2024-09-16T16:39:10.239 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:10.239 INFO:teuthology.orchestra.run.smithi026.stdout:[osd] 2024-09-16T16:39:10.239 INFO:teuthology.orchestra.run.smithi026.stdout: osd journal size = 100 2024-09-16T16:39:10.239 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:10.239 INFO:teuthology.orchestra.run.smithi026.stdout: osd scrub load threshold = 5.0 2024-09-16T16:39:10.240 INFO:teuthology.orchestra.run.smithi026.stdout: osd scrub max interval = 600 2024-09-16T16:39:10.240 INFO:teuthology.orchestra.run.smithi026.stdout: osd mclock profile = high_recovery_ops 2024-09-16T16:39:10.240 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:10.240 INFO:teuthology.orchestra.run.smithi026.stdout: osd recover clone overlap = true 2024-09-16T16:39:10.240 INFO:teuthology.orchestra.run.smithi026.stdout: osd recovery max chunk = 1048576 2024-09-16T16:39:10.240 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:10.240 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug shutdown = true 2024-09-16T16:39:10.240 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug op order = true 2024-09-16T16:39:10.240 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug verify stray on activate = true 2024-09-16T16:39:10.245 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug trim objects = true 2024-09-16T16:39:10.245 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:10.245 INFO:teuthology.orchestra.run.smithi026.stdout: osd open classes on start = true 2024-09-16T16:39:10.245 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug pg log writeout = true 2024-09-16T16:39:10.245 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:10.245 INFO:teuthology.orchestra.run.smithi026.stdout: osd deep scrub update digest min age = 30 2024-09-16T16:39:10.245 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:10.245 INFO:teuthology.orchestra.run.smithi026.stdout: osd map max advance = 10 2024-09-16T16:39:10.245 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:10.245 INFO:teuthology.orchestra.run.smithi026.stdout: journal zero on create = true 2024-09-16T16:39:10.245 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:10.245 INFO:teuthology.orchestra.run.smithi026.stdout: filestore ondisk finisher threads = 3 2024-09-16T16:39:10.246 INFO:teuthology.orchestra.run.smithi026.stdout: filestore apply finisher threads = 3 2024-09-16T16:39:10.246 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:10.246 INFO:teuthology.orchestra.run.smithi026.stdout: bdev debug aio = true 2024-09-16T16:39:10.246 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug misdirected ops = true 2024-09-16T16:39:10.246 INFO:teuthology.orchestra.run.smithi026.stdout: debug monc = 20 2024-09-16T16:39:10.246 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore block size = 96636764160 2024-09-16T16:39:10.246 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore compression algorithm = snappy 2024-09-16T16:39:10.246 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore compression mode = aggressive 2024-09-16T16:39:10.246 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore fsck on mount = True 2024-09-16T16:39:10.246 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore rocksdb cf = False 2024-09-16T16:39:10.246 INFO:teuthology.orchestra.run.smithi026.stdout: crimson alien thread cpu cores = 6-7 2024-09-16T16:39:10.263 INFO:teuthology.orchestra.run.smithi026.stdout: crimson osd obc lru size = 10 2024-09-16T16:39:10.263 INFO:teuthology.orchestra.run.smithi026.stdout: debug bluefs = 20 2024-09-16T16:39:10.263 INFO:teuthology.orchestra.run.smithi026.stdout: debug bluestore = 20 2024-09-16T16:39:10.264 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2024-09-16T16:39:10.264 INFO:teuthology.orchestra.run.smithi026.stdout: debug osd = 20 2024-09-16T16:39:10.264 INFO:teuthology.orchestra.run.smithi026.stdout: debug rocksdb = 10 2024-09-16T16:39:10.264 INFO:teuthology.orchestra.run.smithi026.stdout: err to stderr = True 2024-09-16T16:39:10.264 INFO:teuthology.orchestra.run.smithi026.stdout: log flush on exit = True 2024-09-16T16:39:10.264 INFO:teuthology.orchestra.run.smithi026.stdout: log to file = False 2024-09-16T16:39:10.264 INFO:teuthology.orchestra.run.smithi026.stdout: log to stderr = True 2024-09-16T16:39:10.264 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T16:39:10.264 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd full ratio = 0.9 2024-09-16T16:39:10.264 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd nearfull ratio = 0.8 2024-09-16T16:39:10.264 INFO:teuthology.orchestra.run.smithi026.stdout: osd failsafe full ratio = 0.95 2024-09-16T16:39:10.264 INFO:teuthology.orchestra.run.smithi026.stdout: osd objectstore = bluestore 2024-09-16T16:39:10.264 INFO:teuthology.orchestra.run.smithi026.stdout: osd shutdown pgref assert = True 2024-09-16T16:39:10.265 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:10.265 INFO:teuthology.orchestra.run.smithi026.stdout:[mgr] 2024-09-16T16:39:10.265 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2024-09-16T16:39:10.265 INFO:teuthology.orchestra.run.smithi026.stdout: debug mgr = 20 2024-09-16T16:39:10.265 INFO:teuthology.orchestra.run.smithi026.stdout: debug mon = 20 2024-09-16T16:39:10.265 INFO:teuthology.orchestra.run.smithi026.stdout: debug auth = 20 2024-09-16T16:39:10.265 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min pgs per osd = 4 2024-09-16T16:39:10.265 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min bytes per osd = 10 2024-09-16T16:39:10.265 INFO:teuthology.orchestra.run.smithi026.stdout: mgr/telemetry/nag = false 2024-09-16T16:39:10.265 INFO:teuthology.orchestra.run.smithi026.stdout: mgr stats period = 30 2024-09-16T16:39:10.265 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:10.265 INFO:teuthology.orchestra.run.smithi026.stdout:[mon] 2024-09-16T16:39:10.265 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2024-09-16T16:39:10.265 INFO:teuthology.orchestra.run.smithi026.stdout: debug mon = 20 2024-09-16T16:39:10.266 INFO:teuthology.orchestra.run.smithi026.stdout: debug paxos = 20 2024-09-16T16:39:10.266 INFO:teuthology.orchestra.run.smithi026.stdout: debug auth = 20 2024-09-16T16:39:10.266 INFO:teuthology.orchestra.run.smithi026.stdout: mon data avail warn = 5 2024-09-16T16:39:10.266 INFO:teuthology.orchestra.run.smithi026.stdout: mon mgr mkfs grace = 240 2024-09-16T16:39:10.266 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min pgs per osd = 4 2024-09-16T16:39:10.266 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd reporter subtree level = osd 2024-09-16T16:39:10.266 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd prime pg temp = true 2024-09-16T16:39:10.266 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min bytes per osd = 10 2024-09-16T16:39:10.266 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:10.266 INFO:teuthology.orchestra.run.smithi026.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T16:39:10.266 INFO:teuthology.orchestra.run.smithi026.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T16:39:10.267 INFO:teuthology.orchestra.run.smithi026.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T16:39:10.267 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:10.267 INFO:teuthology.orchestra.run.smithi026.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T16:39:10.267 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T16:39:10.267 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T16:39:10.267 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:10.267 INFO:teuthology.orchestra.run.smithi026.stdout: # 1m isn't quite enough 2024-09-16T16:39:10.267 INFO:teuthology.orchestra.run.smithi026.stdout: mon_down_mkfs_grace = 2m 2024-09-16T16:39:10.267 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:10.267 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_filestore_osds = false 2024-09-16T16:39:10.267 INFO:teuthology.orchestra.run.smithi026.stdout: mon min osdmap epochs = 50 2024-09-16T16:39:10.267 INFO:teuthology.orchestra.run.smithi026.stdout: mon osdmap full prune interval = 2 2024-09-16T16:39:10.267 INFO:teuthology.orchestra.run.smithi026.stdout: mon osdmap full prune min = 15 2024-09-16T16:39:10.267 INFO:teuthology.orchestra.run.smithi026.stdout: mon osdmap full prune txsize = 2 2024-09-16T16:39:10.267 INFO:teuthology.orchestra.run.smithi026.stdout: paxos service trim min = 10 2024-09-16T16:39:10.269 INFO:teuthology.orchestra.run.smithi026.stdout: osd pool default crimson = True 2024-09-16T16:39:10.269 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:10.269 INFO:teuthology.orchestra.run.smithi026.stdout:[client] 2024-09-16T16:39:10.269 INFO:teuthology.orchestra.run.smithi026.stdout: rgw cache enabled = true 2024-09-16T16:39:10.270 INFO:teuthology.orchestra.run.smithi026.stdout: rgw enable ops log = true 2024-09-16T16:39:10.270 INFO:teuthology.orchestra.run.smithi026.stdout: rgw enable usage log = true 2024-09-16T16:39:10.270 INFO:teuthology.orchestra.run.smithi026.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T16:39:10.270 INFO:teuthology.orchestra.run.smithi026.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T16:39:10.270 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.a] 2024-09-16T16:39:10.270 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.0] 2024-09-16T16:39:10.270 INFO:teuthology.orchestra.run.smithi026.stdout: crimson seastar cpu cores = 0-1 2024-09-16T16:39:10.270 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.1] 2024-09-16T16:39:10.270 INFO:teuthology.orchestra.run.smithi026.stdout: crimson seastar cpu cores = 2-3 2024-09-16T16:39:10.270 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.2] 2024-09-16T16:39:10.270 INFO:teuthology.orchestra.run.smithi026.stdout: crimson seastar cpu cores = 4-5 2024-09-16T16:39:10.271 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-16T16:39:10.271 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-16T16:39:10.339 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T16:39:10.339 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T16:39:10.340 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-16T16:39:10.355 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T16:39:10.355 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-16T16:39:10.410 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi026.front.sepia.ceph.com 2024-09-16T16:39:10.411 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T16:39:10.411 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T16:39:10.411 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T16:39:10.487 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T16:39:10.487 DEBUG:teuthology.orchestra.run.smithi026:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T16:39:10.543 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T16:39:10.543 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T16:39:10.543 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T16:39:10.645 INFO:teuthology.orchestra.run.smithi026.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T16:39:10.648 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T16:39:10.648 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T16:39:10.649 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T16:39:10.699 INFO:teuthology.orchestra.run.smithi026.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-16T16:39:10.712 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T16:39:10.713 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi026.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-16T16:39:10.713 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-16T16:39:10.739 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T16:39:10.739 INFO:tasks.ceph:role: osd.0 2024-09-16T16:39:10.739 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi026.front.sepia.ceph.com 2024-09-16T16:39:10.739 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T16:39:10.808 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T16:39:10.808 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T16:39:10.808 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T16:39:10.808 INFO:teuthology.orchestra.run.smithi026.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T16:39:10.808 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T16:39:10.808 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2024-09-16T16:39:10.808 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T16:39:10.808 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T16:39:10.808 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T16:39:10.809 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T16:39:10.840 INFO:teuthology.orchestra.run.smithi026.stdout:Discarding blocks...Done. 2024-09-16T16:39:10.842 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2024-09-16T16:39:10.842 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-16T16:39:10.913 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-16T16:39:10.984 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-16T16:39:11.052 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T16:39:11.052 INFO:tasks.ceph:role: osd.1 2024-09-16T16:39:11.052 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi026.front.sepia.ceph.com 2024-09-16T16:39:11.058 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T16:39:11.121 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T16:39:11.121 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T16:39:11.121 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T16:39:11.122 INFO:teuthology.orchestra.run.smithi026.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T16:39:11.122 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T16:39:11.122 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2024-09-16T16:39:11.122 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T16:39:11.122 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T16:39:11.122 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T16:39:11.122 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T16:39:11.151 INFO:teuthology.orchestra.run.smithi026.stdout:Discarding blocks...Done. 2024-09-16T16:39:11.153 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2024-09-16T16:39:11.153 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-16T16:39:11.225 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-16T16:39:11.296 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-16T16:39:11.365 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T16:39:11.366 INFO:tasks.ceph:role: osd.2 2024-09-16T16:39:11.366 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi026.front.sepia.ceph.com 2024-09-16T16:39:11.366 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T16:39:11.433 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T16:39:11.433 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T16:39:11.433 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T16:39:11.433 INFO:teuthology.orchestra.run.smithi026.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T16:39:11.433 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T16:39:11.433 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2024-09-16T16:39:11.433 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T16:39:11.433 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T16:39:11.433 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T16:39:11.434 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T16:39:11.470 INFO:teuthology.orchestra.run.smithi026.stdout:Discarding blocks...Done. 2024-09-16T16:39:11.470 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2024-09-16T16:39:11.470 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-16T16:39:11.540 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-16T16:39:11.612 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T16:39:11.922 INFO:teuthology.orchestra.run.smithi026.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-16T16:39:11.923 INFO:teuthology.orchestra.run.smithi026.stderr:INFO 2024-09-16 16:39:11,924 seastar - Reactor backend: linux-aio 2024-09-16T16:39:11.934 INFO:teuthology.orchestra.run.smithi026.stderr:==27933==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-16T16:39:12.066 INFO:teuthology.orchestra.run.smithi026.stderr:INFO 2024-09-16 16:39:12,063 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-16T16:39:12.066 INFO:teuthology.orchestra.run.smithi026.stderr:INFO 2024-09-16 16:39:12,063 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 0-1 2024-09-16T16:39:12.066 INFO:teuthology.orchestra.run.smithi026.stderr:DEBUG 2024-09-16 16:39:12,064 [shard 0:main] seastar - reactor::drain 2024-09-16T16:39:12.616 INFO:teuthology.orchestra.run.smithi026.stderr:----------------------------------------------------- 2024-09-16T16:39:12.616 INFO:teuthology.orchestra.run.smithi026.stderr:Suppressions used: 2024-09-16T16:39:12.616 INFO:teuthology.orchestra.run.smithi026.stderr: count bytes template 2024-09-16T16:39:12.616 INFO:teuthology.orchestra.run.smithi026.stderr: 1 8 MallocExtension::Register 2024-09-16T16:39:12.616 INFO:teuthology.orchestra.run.smithi026.stderr:----------------------------------------------------- 2024-09-16T16:39:12.616 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T16:39:12.668 INFO:teuthology.orchestra.run.smithi026.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-16T16:39:12.672 INFO:teuthology.orchestra.run.smithi026.stderr:INFO 2024-09-16 16:39:12,673 seastar - Reactor backend: linux-aio 2024-09-16T16:39:12.680 INFO:teuthology.orchestra.run.smithi026.stderr:==27931==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-16T16:39:12.745 INFO:teuthology.orchestra.run.smithi026.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x5fdcb67 0x5fdd0e1 0x5fdd7df 0x5fdd905 0x6038b24 0x5fb8d06 0x5fb923a 0x5e00fef 0x5e19231 0x71579d2 0x71691a5 0x716a642 0x716a6c7 0x716a7c9 0x716a8c1 0x716aace 0x716acb4 0x716b003 0x716b3a7 0x716b5d7 0x716b841 0x716bb0e 0x716bd2c 0x716bde4 0x716beaa 0x716bf8b 0x5c63e30 0x5c63f3a 0x5c63ff6 0x5c640b2 0x5c6416e 0x5c642bb 0x5c64446 0x5c64807 0x5c7e60a 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T16:39:12.745 INFO:teuthology.orchestra.run.smithi026.stderr:kernel callstack: 2024-09-16T16:39:17.433 INFO:teuthology.orchestra.run.smithi026.stderr:----------------------------------------------------- 2024-09-16T16:39:17.434 INFO:teuthology.orchestra.run.smithi026.stderr:Suppressions used: 2024-09-16T16:39:17.434 INFO:teuthology.orchestra.run.smithi026.stderr: count bytes template 2024-09-16T16:39:17.434 INFO:teuthology.orchestra.run.smithi026.stderr: 1 8 MallocExtension::Register 2024-09-16T16:39:17.434 INFO:teuthology.orchestra.run.smithi026.stderr:----------------------------------------------------- 2024-09-16T16:39:17.434 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T16:39:17.457 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-16T16:39:17.483 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T16:39:17.788 INFO:teuthology.orchestra.run.smithi026.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-16T16:39:17.789 INFO:teuthology.orchestra.run.smithi026.stderr:INFO 2024-09-16 16:39:17,790 seastar - Reactor backend: linux-aio 2024-09-16T16:39:17.795 INFO:teuthology.orchestra.run.smithi026.stderr:==28117==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-16T16:39:17.903 INFO:teuthology.orchestra.run.smithi026.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x4b246 0x5fe9d2d 0x5fe9fdf 0x5fea2b5 0x60389b3 0x6038bb9 0x5fb8d06 0x5fb8ee0 0x5fb946c 0x5e00fef 0x5e19231 0x71579d2 0x71691a5 0x716a642 0x716a6c7 0x716a7c9 0x716a8c1 0x716aace 0x716acb4 0x716b003 0x716b3a7 0x716b5d7 0x716b841 0x716bb0e 0x716bd2c 0x716bde4 0x716beaa 0x716bf8b 0x5c63e30 0x5c63f3a 0x5c63ff6 0x5c640b2 0x5c6416e 0x5c642bb 0x5c64446 0x5c64807 0x5c7e60a 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3962bc0 0x396439a 0x3804c28 0x2958f 0x2963f 0x35ec9b4 2024-09-16T16:39:17.903 INFO:teuthology.orchestra.run.smithi026.stderr:kernel callstack: 2024-09-16T16:39:17.923 INFO:teuthology.orchestra.run.smithi026.stderr:INFO 2024-09-16 16:39:17,923 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-16T16:39:17.923 INFO:teuthology.orchestra.run.smithi026.stderr:INFO 2024-09-16 16:39:17,923 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 2-3 2024-09-16T16:39:17.923 INFO:teuthology.orchestra.run.smithi026.stderr:DEBUG 2024-09-16 16:39:17,924 [shard 0:main] seastar - reactor::drain 2024-09-16T16:39:18.470 INFO:teuthology.orchestra.run.smithi026.stderr:----------------------------------------------------- 2024-09-16T16:39:18.470 INFO:teuthology.orchestra.run.smithi026.stderr:Suppressions used: 2024-09-16T16:39:18.470 INFO:teuthology.orchestra.run.smithi026.stderr: count bytes template 2024-09-16T16:39:18.470 INFO:teuthology.orchestra.run.smithi026.stderr: 1 8 MallocExtension::Register 2024-09-16T16:39:18.470 INFO:teuthology.orchestra.run.smithi026.stderr:----------------------------------------------------- 2024-09-16T16:39:18.470 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T16:39:18.529 INFO:teuthology.orchestra.run.smithi026.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-16T16:39:18.533 INFO:teuthology.orchestra.run.smithi026.stderr:INFO 2024-09-16 16:39:18,534 seastar - Reactor backend: linux-aio 2024-09-16T16:39:18.541 INFO:teuthology.orchestra.run.smithi026.stderr:==28115==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-16T16:39:18.606 INFO:teuthology.orchestra.run.smithi026.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x4ab92 0x3817ea5 0x5e00fd6 0x5e19231 0x71579d2 0x71691a5 0x716a642 0x716a6c7 0x716a7c9 0x716a8c1 0x716aace 0x716acb4 0x716b003 0x716b3a7 0x716b5d7 0x716b841 0x716bb0e 0x716bd2c 0x716bde4 0x716beaa 0x716bf8b 0x5c63e30 0x5c63f3a 0x5c63ff6 0x5c640b2 0x5c6416e 0x5c642bb 0x5c64446 0x5c64807 0x5c7e60a 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T16:39:18.607 INFO:teuthology.orchestra.run.smithi026.stderr:kernel callstack: 2024-09-16T16:39:23.263 INFO:teuthology.orchestra.run.smithi026.stderr:----------------------------------------------------- 2024-09-16T16:39:23.263 INFO:teuthology.orchestra.run.smithi026.stderr:Suppressions used: 2024-09-16T16:39:23.263 INFO:teuthology.orchestra.run.smithi026.stderr: count bytes template 2024-09-16T16:39:23.264 INFO:teuthology.orchestra.run.smithi026.stderr: 1 8 MallocExtension::Register 2024-09-16T16:39:23.264 INFO:teuthology.orchestra.run.smithi026.stderr:----------------------------------------------------- 2024-09-16T16:39:23.264 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T16:39:23.286 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T16:39:23.312 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T16:39:23.615 INFO:teuthology.orchestra.run.smithi026.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-16T16:39:23.616 INFO:teuthology.orchestra.run.smithi026.stderr:INFO 2024-09-16 16:39:23,617 seastar - Reactor backend: linux-aio 2024-09-16T16:39:23.622 INFO:teuthology.orchestra.run.smithi026.stderr:==28301==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-16T16:39:23.730 INFO:teuthology.orchestra.run.smithi026.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x3a9df 0x36084f8 0x362c847 0x362ca9c 0x5fb0d8b 0x5fd278c 0x5fd2a26 0x5fb964c 0x5e00fef 0x5e19231 0x71579d2 0x71691a5 0x716a642 0x716a6c7 0x716a7c9 0x716a8c1 0x716aace 0x716acb4 0x716b003 0x716b3a7 0x716b5d7 0x716b841 0x716bb0e 0x716bd2c 0x716bde4 0x716beaa 0x716bf8b 0x5c63e30 0x5c63f3a 0x5c63ff6 0x5c640b2 0x5c6416e 0x5c642bb 0x5c64446 0x5c64807 0x5c7e60a 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3962bc0 0x396439a 0x3804c28 0x2958f 0x2963f 0x35ec9b4 2024-09-16T16:39:23.731 INFO:teuthology.orchestra.run.smithi026.stderr:kernel callstack: 2024-09-16T16:39:23.752 INFO:teuthology.orchestra.run.smithi026.stderr:INFO 2024-09-16 16:39:23,752 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-16T16:39:23.752 INFO:teuthology.orchestra.run.smithi026.stderr:INFO 2024-09-16 16:39:23,752 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 4-5 2024-09-16T16:39:23.752 INFO:teuthology.orchestra.run.smithi026.stderr:DEBUG 2024-09-16 16:39:23,753 [shard 0:main] seastar - reactor::drain 2024-09-16T16:39:24.286 INFO:teuthology.orchestra.run.smithi026.stderr:----------------------------------------------------- 2024-09-16T16:39:24.286 INFO:teuthology.orchestra.run.smithi026.stderr:Suppressions used: 2024-09-16T16:39:24.286 INFO:teuthology.orchestra.run.smithi026.stderr: count bytes template 2024-09-16T16:39:24.286 INFO:teuthology.orchestra.run.smithi026.stderr: 1 8 MallocExtension::Register 2024-09-16T16:39:24.286 INFO:teuthology.orchestra.run.smithi026.stderr:----------------------------------------------------- 2024-09-16T16:39:24.286 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T16:39:24.348 INFO:teuthology.orchestra.run.smithi026.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-16T16:39:24.352 INFO:teuthology.orchestra.run.smithi026.stderr:INFO 2024-09-16 16:39:24,353 seastar - Reactor backend: linux-aio 2024-09-16T16:39:24.361 INFO:teuthology.orchestra.run.smithi026.stderr:==28299==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-16T16:39:29.066 INFO:teuthology.orchestra.run.smithi026.stderr:----------------------------------------------------- 2024-09-16T16:39:29.067 INFO:teuthology.orchestra.run.smithi026.stderr:Suppressions used: 2024-09-16T16:39:29.067 INFO:teuthology.orchestra.run.smithi026.stderr: count bytes template 2024-09-16T16:39:29.067 INFO:teuthology.orchestra.run.smithi026.stderr: 1 8 MallocExtension::Register 2024-09-16T16:39:29.067 INFO:teuthology.orchestra.run.smithi026.stderr:----------------------------------------------------- 2024-09-16T16:39:29.067 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T16:39:29.090 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T16:39:29.116 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T16:39:29.117 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T16:39:29.117 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-16T16:39:29.183 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T16:39:29.183 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-16T16:39:29.250 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T16:39:29.250 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-16T16:39:29.315 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T16:39:29.316 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-16T16:39:29.381 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T16:39:29.382 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-16T16:39:29.437 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T16:39:29.437 DEBUG:teuthology.orchestra.run.smithi026:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T16:39:29.500 INFO:teuthology.orchestra.run.smithi026.stdout:[mgr.x] 2024-09-16T16:39:29.501 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQAuX+hmcbWFJhAAj8rmf8ENpSAPnUTuCFANpQ== 2024-09-16T16:39:29.501 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.0] 2024-09-16T16:39:29.501 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQAwX+hmUaXBMRAAp2a+rPKiZE/i19Z62MqFkA== 2024-09-16T16:39:29.501 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.1] 2024-09-16T16:39:29.501 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQA2X+hmkFGlKRAAiUonR50Bb6c1Jhc0bPf+qA== 2024-09-16T16:39:29.501 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.2] 2024-09-16T16:39:29.501 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQA8X+hmZv7QHhAAccSbOpTdgZ85g6F7H46R1w== 2024-09-16T16:39:29.501 INFO:teuthology.orchestra.run.smithi026.stdout:[client.0] 2024-09-16T16:39:29.501 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQAuX+hmX2C1KRAAEx9OLeGvqCahsYymJZZWmg== 2024-09-16T16:39:29.502 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T16:39:29.605 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T16:39:29.670 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T16:39:29.732 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T16:39:29.794 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T16:39:29.862 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T16:39:29.863 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-16T16:39:29.890 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T16:39:30.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:30.008+0000 7fe162e30c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:30.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:30.009+0000 7fe162e30c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:30.022 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-16T16:39:30.048 DEBUG:teuthology.orchestra.run.smithi026:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T16:39:30.103 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-16T16:39:30.104 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T16:39:30.104 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T16:39:30.146 INFO:tasks.ceph.mon.a:Started 2024-09-16T16:39:30.147 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-16T16:39:30.147 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-16T16:39:30.147 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-16T16:39:30.149 INFO:tasks.ceph.mgr.x:Started 2024-09-16T16:39:30.149 DEBUG:tasks.ceph:set 0 configs 2024-09-16T16:39:30.149 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T16:39:30.149 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster ceph osd crush tunables default 2024-09-16T16:39:30.233 INFO:tasks.ceph.mgr.x.smithi026.stderr:2024-09-16T16:39:30.232+0000 7f36f5fc5000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:30.233 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-09-16T16:39:30.232+0000 7f4f7eb67c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:30.234 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-09-16T16:39:30.233+0000 7f4f7eb67c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:30.234 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-09-16T16:39:30.233+0000 7f4f7eb67c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:30.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:30.262+0000 7f9a86991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:30.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:30.274+0000 7f9a86991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:30.352 INFO:teuthology.orchestra.run.smithi026.stderr:adjusted tunables profile to default 2024-09-16T16:39:30.364 INFO:tasks.ceph:check_enable_crimson: True 2024-09-16T16:39:30.364 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-09-16T16:39:30.365 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-09-16T16:39:30.439 INFO:tasks.ceph.mgr.x.smithi026.stderr:2024-09-16T16:39:30.439+0000 7f36f5fc5000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:30.439 INFO:tasks.ceph.mgr.x.smithi026.stderr:2024-09-16T16:39:30.439+0000 7f36f5fc5000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:30.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:30.481+0000 7f338df91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:30.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:30.490+0000 7f338df91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:30.597 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-09-16T16:39:30.594+0000 7f4f769f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-09-16T16:39:30.597 INFO:tasks.ceph.mon.a.smithi026.stderr:Please be aware that this feature is experimental, untested, 2024-09-16T16:39:30.597 INFO:tasks.ceph.mon.a.smithi026.stderr:unsupported, and may result in data corruption, data loss, 2024-09-16T16:39:30.597 INFO:tasks.ceph.mon.a.smithi026.stderr:and/or irreparable damage to your cluster. Do not use 2024-09-16T16:39:30.597 INFO:tasks.ceph.mon.a.smithi026.stderr:feature with important data. 2024-09-16T16:39:30.597 INFO:tasks.ceph.mon.a.smithi026.stderr: 2024-09-16T16:39:31.362 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-16T16:39:31.362 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T16:39:31.362 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-16T16:39:31.389 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T16:39:31.389 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-16T16:39:31.455 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T16:39:31.455 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-16T16:39:31.521 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster ceph osd new c51754b3-b226-47a9-942c-c1de35bbfcee 0 2024-09-16T16:39:31.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:31.669+0000 7f4f33151640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:31.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:31.679+0000 7f4f33151640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:31.761 INFO:teuthology.orchestra.run.smithi026.stdout:0 2024-09-16T16:39:31.774 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster ceph osd new dd91df32-3f85-43ce-b983-0b5c1b40059f 1 2024-09-16T16:39:31.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:31.879+0000 7f2615591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:31.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:31.890+0000 7f2615591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:31.971 INFO:teuthology.orchestra.run.smithi026.stdout:1 2024-09-16T16:39:31.982 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster ceph osd new 01c4163c-223f-415b-ba3b-2ab6c501e706 2 2024-09-16T16:39:32.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:32.088+0000 7f3c43e84640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:32.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:32.099+0000 7f3c43e84640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:32.180 INFO:teuthology.orchestra.run.smithi026.stdout:2 2024-09-16T16:39:32.191 DEBUG:teuthology.orchestra.run.smithi026:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-09-16T16:39:32.222 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T16:39:32.222 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-09-16T16:39:32.265 INFO:tasks.ceph.osd.0:Started 2024-09-16T16:39:32.265 DEBUG:teuthology.orchestra.run.smithi026:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-09-16T16:39:32.296 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T16:39:32.296 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-16T16:39:32.341 INFO:tasks.ceph.osd.1:Started 2024-09-16T16:39:32.342 DEBUG:teuthology.orchestra.run.smithi026:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-09-16T16:39:32.380 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T16:39:32.380 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-09-16T16:39:32.423 INFO:tasks.ceph.osd.2:Started 2024-09-16T16:39:32.423 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T16:39:32.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:32.547+0000 7fa694e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:32.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:32.557+0000 7fa694e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:32.630 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:32.630 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":6,"fsid":"3da74dbb-4758-44e9-b920-58aeda43326b","created":"2024-09-16T16:39:30.256938+0000","modified":"2024-09-16T16:39:32.180029+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T16:39:32.180029+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"c51754b3-b226-47a9-942c-c1de35bbfcee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dd91df32-3f85-43ce-b983-0b5c1b40059f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"01c4163c-223f-415b-ba3b-2ab6c501e706","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T16:39:32.642 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-16T16:39:32.642 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T16:39:32.944 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T16:39:33.051 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T16:39:33.047+0000 7f9e2e551640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:33.056 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T16:39:33.056+0000 7f9e2e551640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:33.129 INFO:teuthology.misc.health.smithi026.stdout: 2024-09-16T16:39:33.129 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":6,"fsid":"3da74dbb-4758-44e9-b920-58aeda43326b","created":"2024-09-16T16:39:30.256938+0000","modified":"2024-09-16T16:39:32.180029+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T16:39:32.180029+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"c51754b3-b226-47a9-942c-c1de35bbfcee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dd91df32-3f85-43ce-b983-0b5c1b40059f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"01c4163c-223f-415b-ba3b-2ab6c501e706","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T16:39:33.142 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-16T16:39:36.001 INFO:tasks.ceph.mgr.x.smithi026.stderr:2024-09-16T16:39:36.000+0000 7f3693174640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-16T16:39:39.445 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T16:39:39.563 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T16:39:39.559+0000 7f0f37791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:39.568 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T16:39:39.568+0000 7f0f37791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:39.769 INFO:teuthology.misc.health.smithi026.stdout: 2024-09-16T16:39:39.770 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":9,"fsid":"3da74dbb-4758-44e9-b920-58aeda43326b","created":"2024-09-16T16:39:30.256938+0000","modified":"2024-09-16T16:39:39.005066+0000","last_up_change":"2024-09-16T16:39:39.005066+0000","last_in_change":"2024-09-16T16:39:32.180029+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"c51754b3-b226-47a9-942c-c1de35bbfcee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6805","nonce":637735179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":637735179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6803","nonce":637735179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":637735179}]},"public_addr":"172.21.15.26:6805/637735179","cluster_addr":"172.21.15.26:6804/637735179","heartbeat_back_addr":"172.21.15.26:6803/637735179","heartbeat_front_addr":"172.21.15.26:6802/637735179","state":["exists","up"]},{"osd":1,"uuid":"dd91df32-3f85-43ce-b983-0b5c1b40059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":3366424101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6807","nonce":3366424101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6809","nonce":3366424101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3366424101}]},"public_addr":"172.21.15.26:6806/3366424101","cluster_addr":"172.21.15.26:6807/3366424101","heartbeat_back_addr":"172.21.15.26:6809/3366424101","heartbeat_front_addr":"172.21.15.26:6810/3366424101","state":["exists","up"]},{"osd":2,"uuid":"01c4163c-223f-415b-ba3b-2ab6c501e706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":241914171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6811","nonce":241914171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6813","nonce":241914171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":241914171}]},"public_addr":"172.21.15.26:6808/241914171","cluster_addr":"172.21.15.26:6811/241914171","heartbeat_back_addr":"172.21.15.26:6813/241914171","heartbeat_front_addr":"172.21.15.26:6814/241914171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T16:39:39.781 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-16T16:39:39.781 INFO:tasks.ceph:Creating RBD pool 2024-09-16T16:39:39.781 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-16T16:39:39.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:39.886+0000 7fd903391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:39.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:39.898+0000 7fd903391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:41.007 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'rbd' created 2024-09-16T16:39:41.032 DEBUG:teuthology.orchestra.run.smithi026:> rbd --cluster ceph pool init rbd 2024-09-16T16:39:41.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:41.145+0000 7f6057a65ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:41.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:41.145+0000 7f6057a65ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:41.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:41.152+0000 7f6057a65ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:44.081 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-16T16:39:44.082 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-16T16:39:44.082 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T16:39:44.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:44.194+0000 7f9e45591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:44.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:44.203+0000 7f9e45591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:44.401 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:44.414 INFO:teuthology.orchestra.run.smithi026.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2024-09-16T16:39:44.414 INFO:tasks.ceph_manager:config epoch is 1 2024-09-16T16:39:44.414 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T16:39:44.414 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-16T16:39:44.414 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-16T16:39:44.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:44.523+0000 7f7425b11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:44.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:44.531+0000 7f7425b11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:44.778 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:44.793 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6800","nonce":2031687121},{"type":"v1","addr":"172.21.15.26:6801","nonce":2031687121}]},"active_addr":"172.21.15.26:6801/2031687121","active_change":"2024-09-16T16:39:34.992953+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":3951526608}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":881804135}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":3140538433}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":1368780505}]}]} 2024-09-16T16:39:44.803 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-16T16:39:44.803 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T16:39:44.803 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T16:39:44.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:44.925+0000 7f115d951640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:44.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:44.933+0000 7f115d951640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:45.136 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:45.136 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":14,"fsid":"3da74dbb-4758-44e9-b920-58aeda43326b","created":"2024-09-16T16:39:30.256938+0000","modified":"2024-09-16T16:39:44.028281+0000","last_up_change":"2024-09-16T16:39:39.005066+0000","last_in_change":"2024-09-16T16:39:32.180029+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T16:39:39.015271+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T16:39:40.101450+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c51754b3-b226-47a9-942c-c1de35bbfcee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6805","nonce":637735179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":637735179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6803","nonce":637735179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":637735179}]},"public_addr":"172.21.15.26:6805/637735179","cluster_addr":"172.21.15.26:6804/637735179","heartbeat_back_addr":"172.21.15.26:6803/637735179","heartbeat_front_addr":"172.21.15.26:6802/637735179","state":["exists","up"]},{"osd":1,"uuid":"dd91df32-3f85-43ce-b983-0b5c1b40059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":3366424101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6807","nonce":3366424101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6809","nonce":3366424101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3366424101}]},"public_addr":"172.21.15.26:6806/3366424101","cluster_addr":"172.21.15.26:6807/3366424101","heartbeat_back_addr":"172.21.15.26:6809/3366424101","heartbeat_front_addr":"172.21.15.26:6810/3366424101","state":["exists","up"]},{"osd":2,"uuid":"01c4163c-223f-415b-ba3b-2ab6c501e706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":241914171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6811","nonce":241914171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6813","nonce":241914171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":241914171}]},"public_addr":"172.21.15.26:6808/241914171","cluster_addr":"172.21.15.26:6811/241914171","heartbeat_back_addr":"172.21.15.26:6813/241914171","heartbeat_front_addr":"172.21.15.26:6814/241914171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T16:39:45.148 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T16:39:45.149 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T16:39:45.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:45.258+0000 7f5640afd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:45.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:45.266+0000 7f5640afd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:45.465 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:45.465 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":14,"fsid":"3da74dbb-4758-44e9-b920-58aeda43326b","created":"2024-09-16T16:39:30.256938+0000","modified":"2024-09-16T16:39:44.028281+0000","last_up_change":"2024-09-16T16:39:39.005066+0000","last_in_change":"2024-09-16T16:39:32.180029+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T16:39:39.015271+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T16:39:40.101450+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c51754b3-b226-47a9-942c-c1de35bbfcee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6805","nonce":637735179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":637735179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6803","nonce":637735179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":637735179}]},"public_addr":"172.21.15.26:6805/637735179","cluster_addr":"172.21.15.26:6804/637735179","heartbeat_back_addr":"172.21.15.26:6803/637735179","heartbeat_front_addr":"172.21.15.26:6802/637735179","state":["exists","up"]},{"osd":1,"uuid":"dd91df32-3f85-43ce-b983-0b5c1b40059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":3366424101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6807","nonce":3366424101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6809","nonce":3366424101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3366424101}]},"public_addr":"172.21.15.26:6806/3366424101","cluster_addr":"172.21.15.26:6807/3366424101","heartbeat_back_addr":"172.21.15.26:6809/3366424101","heartbeat_front_addr":"172.21.15.26:6810/3366424101","state":["exists","up"]},{"osd":2,"uuid":"01c4163c-223f-415b-ba3b-2ab6c501e706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":241914171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6811","nonce":241914171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6813","nonce":241914171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":241914171}]},"public_addr":"172.21.15.26:6808/241914171","cluster_addr":"172.21.15.26:6811/241914171","heartbeat_back_addr":"172.21.15.26:6813/241914171","heartbeat_front_addr":"172.21.15.26:6814/241914171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T16:39:45.478 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T16:39:45.479 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T16:39:45.479 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T16:39:45.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:45.594+0000 7fe4aded1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:45.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:45.594+0000 7fd89d4d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:45.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:45.595+0000 7f0ac8d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:45.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:45.604+0000 7fd89d4d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:45.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:45.605+0000 7fe4aded1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:45.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:45.605+0000 7f0ac8d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:45.638 INFO:teuthology.orchestra.run.smithi026.stdout:34359738375 2024-09-16T16:39:45.639 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T16:39:45.640 INFO:teuthology.orchestra.run.smithi026.stdout:34359738375 2024-09-16T16:39:45.640 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T16:39:45.642 INFO:teuthology.orchestra.run.smithi026.stdout:38654705670 2024-09-16T16:39:45.642 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T16:39:45.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:45.750+0000 7fb6ebf51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:45.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:45.762+0000 7fb6ebf51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:45.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:45.796+0000 7fc7a7f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:45.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:45.796+0000 7fdb8d591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:45.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:45.806+0000 7fdb8d591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:45.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:45.806+0000 7fc7a7f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:45.967 INFO:teuthology.orchestra.run.smithi026.stdout:34359738375 2024-09-16T16:39:45.981 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738375 for osd.0 2024-09-16T16:39:45.982 DEBUG:teuthology.parallel:result is None 2024-09-16T16:39:46.006 INFO:teuthology.orchestra.run.smithi026.stdout:38654705670 2024-09-16T16:39:46.013 INFO:teuthology.orchestra.run.smithi026.stdout:34359738375 2024-09-16T16:39:46.020 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705670 for osd.2 2024-09-16T16:39:46.020 DEBUG:teuthology.parallel:result is None 2024-09-16T16:39:46.028 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738375 for osd.1 2024-09-16T16:39:46.029 DEBUG:teuthology.parallel:result is None 2024-09-16T16:39:46.029 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T16:39:46.029 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T16:39:46.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:46.179+0000 7f0098f51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:46.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:46.187+0000 7f0098f51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:46.383 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:46.384 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-16T16:39:46.396 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-09-16T16:39:45.629834+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":81004,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034516,"statfs":{"total":289910292480,"available":289827344384,"internally_reserved":0,"allocated":487424,"data_stored":1269292,"data_compressed":59980,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-16T16:39:44.046081+0000","last_change":"2024-09-16T16:39:44.053256+0000","last_active":"2024-09-16T16:39:44.046081+0000","last_peered":"2024-09-16T16:39:44.046081+0000","last_clean":"2024-09-16T16:39:44.046081+0000","last_became_active":"2024-09-16T16:39:42.085202+0000","last_became_peered":"2024-09-16T16:39:42.085202+0000","last_unstale":"2024-09-16T16:39:44.046081+0000","last_undegraded":"2024-09-16T16:39:44.046081+0000","last_fullsized":"2024-09-16T16:39:44.046081+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-16T16:39:44.057415+0000","last_change":"2024-09-16T16:39:44.063464+0000","last_active":"2024-09-16T16:39:44.057415+0000","last_peered":"2024-09-16T16:39:44.057415+0000","last_clean":"2024-09-16T16:39:44.057415+0000","last_became_active":"2024-09-16T16:39:42.115450+0000","last_became_peered":"2024-09-16T16:39:42.115450+0000","last_unstale":"2024-09-16T16:39:44.057415+0000","last_undegraded":"2024-09-16T16:39:44.057415+0000","last_fullsized":"2024-09-16T16:39:44.057415+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-16T16:39:44.055766+0000","last_change":"2024-09-16T16:39:44.064037+0000","last_active":"2024-09-16T16:39:44.055766+0000","last_peered":"2024-09-16T16:39:44.055766+0000","last_clean":"2024-09-16T16:39:44.055766+0000","last_became_active":"2024-09-16T16:39:42.129388+0000","last_became_peered":"2024-09-16T16:39:42.129388+0000","last_unstale":"2024-09-16T16:39:44.055766+0000","last_undegraded":"2024-09-16T16:39:44.055766+0000","last_fullsized":"2024-09-16T16:39:44.055766+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-16T16:39:44.059677+0000","last_change":"2024-09-16T16:39:44.064988+0000","last_active":"2024-09-16T16:39:44.059677+0000","last_peered":"2024-09-16T16:39:44.059677+0000","last_clean":"2024-09-16T16:39:44.059677+0000","last_became_active":"2024-09-16T16:39:42.135711+0000","last_became_peered":"2024-09-16T16:39:42.135711+0000","last_unstale":"2024-09-16T16:39:44.059677+0000","last_undegraded":"2024-09-16T16:39:44.059677+0000","last_fullsized":"2024-09-16T16:39:44.059677+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-16T16:39:44.053534+0000","last_change":"2024-09-16T16:39:44.053534+0000","last_active":"2024-09-16T16:39:44.053534+0000","last_peered":"2024-09-16T16:39:44.053534+0000","last_clean":"2024-09-16T16:39:44.053534+0000","last_became_active":"2024-09-16T16:39:42.116188+0000","last_became_peered":"2024-09-16T16:39:42.116188+0000","last_unstale":"2024-09-16T16:39:44.053534+0000","last_undegraded":"2024-09-16T16:39:44.053534+0000","last_fullsized":"2024-09-16T16:39:44.053534+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-16T16:39:44.059684+0000","last_change":"2024-09-16T16:39:44.062407+0000","last_active":"2024-09-16T16:39:44.059684+0000","last_peered":"2024-09-16T16:39:44.059684+0000","last_clean":"2024-09-16T16:39:44.059684+0000","last_became_active":"2024-09-16T16:39:42.148215+0000","last_became_peered":"2024-09-16T16:39:42.148215+0000","last_unstale":"2024-09-16T16:39:44.059684+0000","last_undegraded":"2024-09-16T16:39:44.059684+0000","last_fullsized":"2024-09-16T16:39:44.059684+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-16T16:39:44.053936+0000","last_change":"2024-09-16T16:39:44.058579+0000","last_active":"2024-09-16T16:39:44.053936+0000","last_peered":"2024-09-16T16:39:44.053936+0000","last_clean":"2024-09-16T16:39:44.053936+0000","last_became_active":"2024-09-16T16:39:42.133685+0000","last_became_peered":"2024-09-16T16:39:42.133685+0000","last_unstale":"2024-09-16T16:39:44.053936+0000","last_undegraded":"2024-09-16T16:39:44.053936+0000","last_fullsized":"2024-09-16T16:39:44.053936+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-16T16:39:44.053399+0000","last_change":"2024-09-16T16:39:44.064255+0000","last_active":"2024-09-16T16:39:44.053399+0000","last_peered":"2024-09-16T16:39:44.053399+0000","last_clean":"2024-09-16T16:39:44.053399+0000","last_became_active":"2024-09-16T16:39:42.094003+0000","last_became_peered":"2024-09-16T16:39:42.094003+0000","last_unstale":"2024-09-16T16:39:44.053399+0000","last_undegraded":"2024-09-16T16:39:44.053399+0000","last_fullsized":"2024-09-16T16:39:44.053399+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":84,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-16T16:39:44.049478+0000","last_change":"2024-09-16T16:39:41.078026+0000","last_active":"2024-09-16T16:39:44.049478+0000","last_peered":"2024-09-16T16:39:44.049478+0000","last_clean":"2024-09-16T16:39:44.049478+0000","last_became_active":"2024-09-16T16:39:41.066265+0000","last_became_peered":"2024-09-16T16:39:41.066265+0000","last_unstale":"2024-09-16T16:39:44.049478+0000","last_undegraded":"2024-09-16T16:39:44.049478+0000","last_fullsized":"2024-09-16T16:39:44.049478+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:40.005960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:40.005960+0000","last_clean_scrub_stamp":"2024-09-16T16:39:40.005960+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29990,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26948,"kb_used_data":116,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344892,"statfs":{"total":96636764160,"available":96609169408,"internally_reserved":0,"allocated":118784,"data_stored":29506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27028,"kb_used_data":180,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344812,"statfs":{"total":96636764160,"available":96609087488,"internally_reserved":0,"allocated":184320,"data_stored":619893,"data_compressed":29990,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27028,"kb_used_data":180,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344812,"statfs":{"total":96636764160,"available":96609087488,"internally_reserved":0,"allocated":184320,"data_stored":619893,"data_compressed":29990,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29990,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T16:39:46.397 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T16:39:46.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:46.507+0000 7faebfb11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:46.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:46.516+0000 7faebfb11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:46.713 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:46.713 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-16T16:39:46.727 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-09-16T16:39:45.629834+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":81004,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034516,"statfs":{"total":289910292480,"available":289827344384,"internally_reserved":0,"allocated":487424,"data_stored":1269292,"data_compressed":59980,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-16T16:39:44.046081+0000","last_change":"2024-09-16T16:39:44.053256+0000","last_active":"2024-09-16T16:39:44.046081+0000","last_peered":"2024-09-16T16:39:44.046081+0000","last_clean":"2024-09-16T16:39:44.046081+0000","last_became_active":"2024-09-16T16:39:42.085202+0000","last_became_peered":"2024-09-16T16:39:42.085202+0000","last_unstale":"2024-09-16T16:39:44.046081+0000","last_undegraded":"2024-09-16T16:39:44.046081+0000","last_fullsized":"2024-09-16T16:39:44.046081+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-16T16:39:44.057415+0000","last_change":"2024-09-16T16:39:44.063464+0000","last_active":"2024-09-16T16:39:44.057415+0000","last_peered":"2024-09-16T16:39:44.057415+0000","last_clean":"2024-09-16T16:39:44.057415+0000","last_became_active":"2024-09-16T16:39:42.115450+0000","last_became_peered":"2024-09-16T16:39:42.115450+0000","last_unstale":"2024-09-16T16:39:44.057415+0000","last_undegraded":"2024-09-16T16:39:44.057415+0000","last_fullsized":"2024-09-16T16:39:44.057415+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-16T16:39:44.055766+0000","last_change":"2024-09-16T16:39:44.064037+0000","last_active":"2024-09-16T16:39:44.055766+0000","last_peered":"2024-09-16T16:39:44.055766+0000","last_clean":"2024-09-16T16:39:44.055766+0000","last_became_active":"2024-09-16T16:39:42.129388+0000","last_became_peered":"2024-09-16T16:39:42.129388+0000","last_unstale":"2024-09-16T16:39:44.055766+0000","last_undegraded":"2024-09-16T16:39:44.055766+0000","last_fullsized":"2024-09-16T16:39:44.055766+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-16T16:39:44.059677+0000","last_change":"2024-09-16T16:39:44.064988+0000","last_active":"2024-09-16T16:39:44.059677+0000","last_peered":"2024-09-16T16:39:44.059677+0000","last_clean":"2024-09-16T16:39:44.059677+0000","last_became_active":"2024-09-16T16:39:42.135711+0000","last_became_peered":"2024-09-16T16:39:42.135711+0000","last_unstale":"2024-09-16T16:39:44.059677+0000","last_undegraded":"2024-09-16T16:39:44.059677+0000","last_fullsized":"2024-09-16T16:39:44.059677+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-16T16:39:44.053534+0000","last_change":"2024-09-16T16:39:44.053534+0000","last_active":"2024-09-16T16:39:44.053534+0000","last_peered":"2024-09-16T16:39:44.053534+0000","last_clean":"2024-09-16T16:39:44.053534+0000","last_became_active":"2024-09-16T16:39:42.116188+0000","last_became_peered":"2024-09-16T16:39:42.116188+0000","last_unstale":"2024-09-16T16:39:44.053534+0000","last_undegraded":"2024-09-16T16:39:44.053534+0000","last_fullsized":"2024-09-16T16:39:44.053534+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-16T16:39:44.059684+0000","last_change":"2024-09-16T16:39:44.062407+0000","last_active":"2024-09-16T16:39:44.059684+0000","last_peered":"2024-09-16T16:39:44.059684+0000","last_clean":"2024-09-16T16:39:44.059684+0000","last_became_active":"2024-09-16T16:39:42.148215+0000","last_became_peered":"2024-09-16T16:39:42.148215+0000","last_unstale":"2024-09-16T16:39:44.059684+0000","last_undegraded":"2024-09-16T16:39:44.059684+0000","last_fullsized":"2024-09-16T16:39:44.059684+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-16T16:39:44.053936+0000","last_change":"2024-09-16T16:39:44.058579+0000","last_active":"2024-09-16T16:39:44.053936+0000","last_peered":"2024-09-16T16:39:44.053936+0000","last_clean":"2024-09-16T16:39:44.053936+0000","last_became_active":"2024-09-16T16:39:42.133685+0000","last_became_peered":"2024-09-16T16:39:42.133685+0000","last_unstale":"2024-09-16T16:39:44.053936+0000","last_undegraded":"2024-09-16T16:39:44.053936+0000","last_fullsized":"2024-09-16T16:39:44.053936+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-16T16:39:44.053399+0000","last_change":"2024-09-16T16:39:44.064255+0000","last_active":"2024-09-16T16:39:44.053399+0000","last_peered":"2024-09-16T16:39:44.053399+0000","last_clean":"2024-09-16T16:39:44.053399+0000","last_became_active":"2024-09-16T16:39:42.094003+0000","last_became_peered":"2024-09-16T16:39:42.094003+0000","last_unstale":"2024-09-16T16:39:44.053399+0000","last_undegraded":"2024-09-16T16:39:44.053399+0000","last_fullsized":"2024-09-16T16:39:44.053399+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":84,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-16T16:39:44.049478+0000","last_change":"2024-09-16T16:39:41.078026+0000","last_active":"2024-09-16T16:39:44.049478+0000","last_peered":"2024-09-16T16:39:44.049478+0000","last_clean":"2024-09-16T16:39:44.049478+0000","last_became_active":"2024-09-16T16:39:41.066265+0000","last_became_peered":"2024-09-16T16:39:41.066265+0000","last_unstale":"2024-09-16T16:39:44.049478+0000","last_undegraded":"2024-09-16T16:39:44.049478+0000","last_fullsized":"2024-09-16T16:39:44.049478+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:40.005960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:40.005960+0000","last_clean_scrub_stamp":"2024-09-16T16:39:40.005960+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29990,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26948,"kb_used_data":116,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344892,"statfs":{"total":96636764160,"available":96609169408,"internally_reserved":0,"allocated":118784,"data_stored":29506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27028,"kb_used_data":180,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344812,"statfs":{"total":96636764160,"available":96609087488,"internally_reserved":0,"allocated":184320,"data_stored":619893,"data_compressed":29990,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27028,"kb_used_data":180,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344812,"statfs":{"total":96636764160,"available":96609087488,"internally_reserved":0,"allocated":184320,"data_stored":619893,"data_compressed":29990,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29990,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T16:39:46.728 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T16:39:46.728 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T16:39:46.728 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-16T16:39:46.728 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T16:39:46.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:46.836+0000 7fe360b51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:46.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:39:46.843+0000 7fe360b51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:39:47.080 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:39:47.081 INFO:teuthology.orchestra.run.smithi026.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T16:39:47.093 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-16T16:39:47.094 INFO:teuthology.run_tasks:Running task workunit... 2024-09-16T16:39:47.107 INFO:tasks.workunit:Pulling workunits from ref 0325fd3c11a225c0fd9eb38fab88aad1370ad779 2024-09-16T16:39:47.108 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-16T16:39:47.108 DEBUG:teuthology.orchestra.run.smithi026:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-16T16:39:47.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T16:39:47.126 INFO:teuthology.orchestra.run.smithi026.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T16:39:47.127 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T16:39:47.186 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-16T16:39:47.186 DEBUG:teuthology.orchestra.run.smithi026:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-16T16:39:47.242 INFO:tasks.workunit:timeout=3h 2024-09-16T16:39:47.242 INFO:tasks.workunit:cleanup=True 2024-09-16T16:39:47.243 DEBUG:teuthology.orchestra.run.smithi026:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 0325fd3c11a225c0fd9eb38fab88aad1370ad779 2024-09-16T16:39:47.300 INFO:tasks.workunit.client.0.smithi026.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-16T16:40:39.023 INFO:tasks.workunit.client.0.smithi026.stderr:Updating files: 94% (11440/12088) Updating files: 95% (11484/12088) Updating files: 96% (11605/12088) Updating files: 97% (11726/12088) Updating files: 98% (11847/12088) Updating files: 99% (11968/12088) Updating files: 100% (12088/12088) Updating files: 100% (12088/12088), done. 2024-09-16T16:40:39.533 INFO:tasks.workunit.client.0.smithi026.stderr:Note: switching to '0325fd3c11a225c0fd9eb38fab88aad1370ad779'. 2024-09-16T16:40:39.533 INFO:tasks.workunit.client.0.smithi026.stderr: 2024-09-16T16:40:39.533 INFO:tasks.workunit.client.0.smithi026.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-16T16:40:39.533 INFO:tasks.workunit.client.0.smithi026.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-16T16:40:39.533 INFO:tasks.workunit.client.0.smithi026.stderr:state without impacting any branches by switching back to a branch. 2024-09-16T16:40:39.533 INFO:tasks.workunit.client.0.smithi026.stderr: 2024-09-16T16:40:39.533 INFO:tasks.workunit.client.0.smithi026.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-16T16:40:39.534 INFO:tasks.workunit.client.0.smithi026.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-16T16:40:39.534 INFO:tasks.workunit.client.0.smithi026.stderr: 2024-09-16T16:40:39.534 INFO:tasks.workunit.client.0.smithi026.stderr: git switch -c 2024-09-16T16:40:39.534 INFO:tasks.workunit.client.0.smithi026.stderr: 2024-09-16T16:40:39.534 INFO:tasks.workunit.client.0.smithi026.stderr:Or undo this operation with: 2024-09-16T16:40:39.534 INFO:tasks.workunit.client.0.smithi026.stderr: 2024-09-16T16:40:39.534 INFO:tasks.workunit.client.0.smithi026.stderr: git switch - 2024-09-16T16:40:39.534 INFO:tasks.workunit.client.0.smithi026.stderr: 2024-09-16T16:40:39.534 INFO:tasks.workunit.client.0.smithi026.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-16T16:40:39.534 INFO:tasks.workunit.client.0.smithi026.stderr: 2024-09-16T16:40:39.535 INFO:tasks.workunit.client.0.smithi026.stderr:HEAD is now at 0325fd3c11a Merge pull request #58530 from zdover23/wip-doc-2024-07-11-radosgw-s3-authentication 2024-09-16T16:40:39.538 DEBUG:teuthology.orchestra.run.smithi026:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-16T16:40:39.605 INFO:tasks.workunit.client.0.smithi026.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-16T16:40:39.607 INFO:tasks.workunit.client.0.smithi026.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T16:40:39.607 INFO:tasks.workunit.client.0.smithi026.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-16T16:40:40.264 INFO:tasks.workunit.client.0.smithi026.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-16T16:40:40.332 INFO:tasks.workunit.client.0.smithi026.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-16T16:40:40.374 INFO:tasks.workunit.client.0.smithi026.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T16:40:40.376 INFO:tasks.workunit.client.0.smithi026.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T16:40:40.376 INFO:tasks.workunit.client.0.smithi026.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-16T16:40:40.416 INFO:tasks.workunit.client.0.smithi026.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T16:40:40.419 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T16:40:40.419 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-16T16:40:40.475 INFO:tasks.workunit:Running workunits matching rbd/test_lock_fence.sh on client.0... 2024-09-16T16:40:40.477 INFO:tasks.workunit:Running workunit rbd/test_lock_fence.sh... 2024-09-16T16:40:40.477 DEBUG:teuthology.orchestra.run.smithi026:workunit test rbd/test_lock_fence.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0325fd3c11a225c0fd9eb38fab88aad1370ad779 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_lock_fence.sh 2024-09-16T16:40:40.536 INFO:tasks.workunit.client.0.smithi026.stderr:+ IMAGE=rbdrw-image 2024-09-16T16:40:40.536 INFO:tasks.workunit.client.0.smithi026.stderr:+ LOCKID=rbdrw 2024-09-16T16:40:40.536 INFO:tasks.workunit.client.0.smithi026.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_lock_fence.sh 2024-09-16T16:40:40.536 INFO:tasks.workunit.client.0.smithi026.stderr:+ RELPATH=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/librbd 2024-09-16T16:40:40.537 INFO:tasks.workunit.client.0.smithi026.stderr:+ RBDRW=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/librbd/rbdrw.py 2024-09-16T16:40:40.537 INFO:tasks.workunit.client.0.smithi026.stderr:+ rbd create rbdrw-image --size 10 --image-format 2 --image-shared 2024-09-16T16:40:40.581 INFO:tasks.workunit.client.0.smithi026.stderr:2024-09-16T16:40:40.580+0000 7fa4d280dac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:40:40.581 INFO:tasks.workunit.client.0.smithi026.stderr:2024-09-16T16:40:40.580+0000 7fa4d280dac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:40:40.586 INFO:tasks.workunit.client.0.smithi026.stderr:2024-09-16T16:40:40.586+0000 7fa4d280dac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:40:40.673 INFO:tasks.workunit.client.0.smithi026.stderr:+ iochild=30590 2024-09-16T16:40:40.673 INFO:tasks.workunit.client.0.smithi026.stderr:+ LOCKS='[]' 2024-09-16T16:40:40.673 INFO:tasks.workunit.client.0.smithi026.stderr:+ python3 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/librbd/rbdrw.py rbdrw-image rbdrw 2024-09-16T16:40:40.673 INFO:tasks.workunit.client.0.smithi026.stderr:+ '[' '[]' == '[]' ']' 2024-09-16T16:40:40.673 INFO:tasks.workunit.client.0.smithi026.stderr:++ rbd lock list rbdrw-image --format json 2024-09-16T16:40:40.717 INFO:tasks.workunit.client.0.smithi026.stderr:2024-09-16T16:40:40.716+0000 7f8248a18ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:40:40.717 INFO:tasks.workunit.client.0.smithi026.stderr:2024-09-16T16:40:40.716+0000 7f8248a18ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:40:40.723 INFO:tasks.workunit.client.0.smithi026.stderr:2024-09-16T16:40:40.723+0000 7f8248a18ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:40:40.855 INFO:tasks.workunit.client.0.smithi026.stderr:+ LOCKS='[]' 2024-09-16T16:40:40.856 INFO:tasks.workunit.client.0.smithi026.stderr:+ sleep 1 2024-09-16T16:40:41.857 INFO:tasks.workunit.client.0.smithi026.stderr:+ '[' '[]' == '[]' ']' 2024-09-16T16:40:41.857 INFO:tasks.workunit.client.0.smithi026.stderr:++ rbd lock list rbdrw-image --format json 2024-09-16T16:40:41.907 INFO:tasks.workunit.client.0.smithi026.stderr:2024-09-16T16:40:41.907+0000 7ffbd8c65ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:40:41.907 INFO:tasks.workunit.client.0.smithi026.stderr:2024-09-16T16:40:41.907+0000 7ffbd8c65ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:40:41.913 INFO:tasks.workunit.client.0.smithi026.stderr:2024-09-16T16:40:41.913+0000 7ffbd8c65ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:40:42.066 INFO:tasks.workunit.client.0.smithi026.stderr:+ LOCKS='[{"id":"rbdrw","locker":"client.4194","address":"172.21.15.26:0/364689746"}]' 2024-09-16T16:40:42.066 INFO:tasks.workunit.client.0.smithi026.stderr:+ sleep 1 2024-09-16T16:40:43.068 INFO:tasks.workunit.client.0.smithi026.stderr:+ '[' '[{"id":"rbdrw","locker":"client.4194","address":"172.21.15.26:0/364689746"}]' == '[]' ']' 2024-09-16T16:40:43.068 INFO:tasks.workunit.client.0.smithi026.stderr:++ rbd lock list rbdrw-image 2024-09-16T16:40:43.068 INFO:tasks.workunit.client.0.smithi026.stderr:++ tail -1 2024-09-16T16:40:43.068 INFO:tasks.workunit.client.0.smithi026.stderr:++ awk '{print $NF;}' 2024-09-16T16:40:43.113 INFO:tasks.workunit.client.0.smithi026.stderr:2024-09-16T16:40:43.113+0000 7fdb9e7d9ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:40:43.113 INFO:tasks.workunit.client.0.smithi026.stderr:2024-09-16T16:40:43.113+0000 7fdb9e7d9ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:40:43.122 INFO:tasks.workunit.client.0.smithi026.stderr:2024-09-16T16:40:43.122+0000 7fdb9e7d9ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:40:43.263 INFO:tasks.workunit.client.0.smithi026.stderr:+ clientaddr=172.21.15.26:0/364689746 2024-09-16T16:40:43.263 INFO:tasks.workunit.client.0.smithi026.stderr:++ rbd lock list rbdrw-image 2024-09-16T16:40:43.263 INFO:tasks.workunit.client.0.smithi026.stderr:++ tail -1 2024-09-16T16:40:43.263 INFO:tasks.workunit.client.0.smithi026.stderr:++ awk '{print $1;}' 2024-09-16T16:40:43.310 INFO:tasks.workunit.client.0.smithi026.stderr:2024-09-16T16:40:43.310+0000 7fa1d2841ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:40:43.310 INFO:tasks.workunit.client.0.smithi026.stderr:2024-09-16T16:40:43.310+0000 7fa1d2841ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:40:43.316 INFO:tasks.workunit.client.0.smithi026.stderr:2024-09-16T16:40:43.316+0000 7fa1d2841ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:40:43.467 INFO:tasks.workunit.client.0.smithi026.stderr:+ clientid=client.4194 2024-09-16T16:40:43.468 INFO:tasks.workunit.client.0.smithi026.stdout:clientaddr: 172.21.15.26:0/364689746 2024-09-16T16:40:43.468 INFO:tasks.workunit.client.0.smithi026.stdout:clientid: client.4194 2024-09-16T16:40:43.468 INFO:tasks.workunit.client.0.smithi026.stderr:+ echo 'clientaddr: 172.21.15.26:0/364689746' 2024-09-16T16:40:43.468 INFO:tasks.workunit.client.0.smithi026.stderr:+ echo 'clientid: client.4194' 2024-09-16T16:40:43.468 INFO:tasks.workunit.client.0.smithi026.stderr:+ ceph osd blocklist add 172.21.15.26:0/364689746 2024-09-16T16:40:43.584 INFO:tasks.workunit.client.0.smithi026.stderr:2024-09-16T16:40:43.577+0000 7f11d3111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:40:43.589 INFO:tasks.workunit.client.0.smithi026.stderr:2024-09-16T16:40:43.589+0000 7f11d3111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:40:45.275 INFO:tasks.workunit.client.0.smithi026.stderr:blocklisting 172.21.15.26:0/364689746 until 2024-09-16T17:40:44.410768+0000 (3600 sec) 2024-09-16T16:40:45.295 INFO:tasks.workunit.client.0.smithi026.stderr:+ wait 30590 2024-09-16T16:40:45.295 INFO:tasks.workunit.client.0.smithi026.stderr:+ rbdrw_exitcode=108 2024-09-16T16:40:45.295 INFO:tasks.workunit.client.0.smithi026.stderr:+ '[' 108 '!=' 108 ']' 2024-09-16T16:40:45.296 INFO:tasks.workunit.client.0.smithi026.stdout:rbdrw stopped with ESHUTDOWN 2024-09-16T16:40:45.296 INFO:tasks.workunit.client.0.smithi026.stderr:+ echo 'rbdrw stopped with ESHUTDOWN' 2024-09-16T16:40:45.296 INFO:tasks.workunit.client.0.smithi026.stderr:+ set -e 2024-09-16T16:40:45.296 INFO:tasks.workunit.client.0.smithi026.stderr:+ ceph osd blocklist rm 172.21.15.26:0/364689746 2024-09-16T16:40:45.398 INFO:tasks.workunit.client.0.smithi026.stderr:2024-09-16T16:40:45.392+0000 7f6c02efd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:40:45.402 INFO:tasks.workunit.client.0.smithi026.stderr:2024-09-16T16:40:45.402+0000 7f6c02efd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:40:46.425 INFO:tasks.workunit.client.0.smithi026.stderr:172.21.15.26:0/364689746 isn't blocklisted 2024-09-16T16:40:46.437 INFO:tasks.workunit.client.0.smithi026.stderr:+ rbd lock remove rbdrw-image rbdrw client.4194 2024-09-16T16:40:46.478 INFO:tasks.workunit.client.0.smithi026.stderr:2024-09-16T16:40:46.478+0000 7f61a6812ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:40:46.478 INFO:tasks.workunit.client.0.smithi026.stderr:2024-09-16T16:40:46.478+0000 7f61a6812ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:40:46.484 INFO:tasks.workunit.client.0.smithi026.stderr:2024-09-16T16:40:46.484+0000 7f61a6812ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:40:47.356 INFO:tasks.workunit.client.0.smithi026.stderr:+ sleep 30 2024-09-16T16:41:17.357 INFO:tasks.workunit.client.0.smithi026.stderr:+ rbd rm rbdrw-image 2024-09-16T16:41:17.405 INFO:tasks.workunit.client.0.smithi026.stderr:2024-09-16T16:41:17.404+0000 7f15a3040ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:41:17.405 INFO:tasks.workunit.client.0.smithi026.stderr:2024-09-16T16:41:17.404+0000 7f15a3040ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:41:17.410 INFO:tasks.workunit.client.0.smithi026.stderr:2024-09-16T16:41:17.410+0000 7f15a3040ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:41:17.901 INFO:tasks.workunit.client.0.smithi026.stderr: Removing image: 33% complete... Removing image: 66% complete... Removing image: 100% complete...done. 2024-09-16T16:41:17.908 INFO:tasks.workunit.client.0.smithi026.stdout:OK 2024-09-16T16:41:17.909 INFO:tasks.workunit.client.0.smithi026.stderr:+ echo OK 2024-09-16T16:41:17.910 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T16:41:17.910 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T16:41:17.976 INFO:tasks.workunit:Stopping ['rbd/test_lock_fence.sh'] on client.0... 2024-09-16T16:41:17.977 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-16T16:41:18.321 DEBUG:teuthology.parallel:result is None 2024-09-16T16:41:18.321 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T16:41:18.347 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T16:41:18.347 DEBUG:teuthology.orchestra.run.smithi026:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T16:41:18.409 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T16:41:18.410 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T16:41:18.422 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T16:41:18.422 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T16:41:18.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:41:18.560+0000 7f520e991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:41:18.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:41:18.568+0000 7f520e991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:41:18.761 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:41:18.761 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-16T16:41:18.773 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":61,"stamp":"2024-09-16T16:41:17.017047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594499,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":412,"num_read_kb":735,"num_write":254,"num_write_kb":1783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":232,"ondisk_log_size":232,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":81084,"kb_used_data":588,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034436,"statfs":{"total":289910292480,"available":289827262464,"internally_reserved":0,"allocated":602112,"data_stored":1309391,"data_compressed":59980,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4112,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":359,"num_read_kb":689,"num_write":173,"num_write_kb":675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003099"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.304933+0000","last_change":"2024-09-16T16:39:44.053256+0000","last_active":"2024-09-16T16:40:47.304933+0000","last_peered":"2024-09-16T16:40:47.304933+0000","last_clean":"2024-09-16T16:40:47.304933+0000","last_became_active":"2024-09-16T16:39:42.085202+0000","last_became_peered":"2024-09-16T16:39:42.085202+0000","last_unstale":"2024-09-16T16:40:47.304933+0000","last_undegraded":"2024-09-16T16:40:47.304933+0000","last_fullsized":"2024-09-16T16:40:47.304933+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.311794+0000","last_change":"2024-09-16T16:39:44.063464+0000","last_active":"2024-09-16T16:40:47.311794+0000","last_peered":"2024-09-16T16:40:47.311794+0000","last_clean":"2024-09-16T16:40:47.311794+0000","last_became_active":"2024-09-16T16:39:42.115450+0000","last_became_peered":"2024-09-16T16:39:42.115450+0000","last_unstale":"2024-09-16T16:40:47.311794+0000","last_undegraded":"2024-09-16T16:40:47.311794+0000","last_fullsized":"2024-09-16T16:40:47.311794+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"14'169","reported_seq":375,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.307691+0000","last_change":"2024-09-16T16:39:44.064037+0000","last_active":"2024-09-16T16:40:47.307691+0000","last_peered":"2024-09-16T16:40:47.307691+0000","last_clean":"2024-09-16T16:40:47.307691+0000","last_became_active":"2024-09-16T16:39:42.129388+0000","last_became_peered":"2024-09-16T16:39:42.129388+0000","last_unstale":"2024-09-16T16:40:47.307691+0000","last_undegraded":"2024-09-16T16:40:47.307691+0000","last_fullsized":"2024-09-16T16:40:47.307691+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4112,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":187,"num_read_kb":674,"num_write":169,"num_write_kb":673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"18'8","reported_seq":93,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.335859+0000","last_change":"2024-09-16T16:39:44.064988+0000","last_active":"2024-09-16T16:40:47.335859+0000","last_peered":"2024-09-16T16:40:47.335859+0000","last_clean":"2024-09-16T16:40:47.335859+0000","last_became_active":"2024-09-16T16:39:42.135711+0000","last_became_peered":"2024-09-16T16:39:42.135711+0000","last_unstale":"2024-09-16T16:40:47.335859+0000","last_undegraded":"2024-09-16T16:40:47.335859+0000","last_fullsized":"2024-09-16T16:40:47.335859+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":163,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":38,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.305274+0000","last_change":"2024-09-16T16:39:44.053534+0000","last_active":"2024-09-16T16:40:47.305274+0000","last_peered":"2024-09-16T16:40:47.305274+0000","last_clean":"2024-09-16T16:40:47.305274+0000","last_became_active":"2024-09-16T16:39:42.116188+0000","last_became_peered":"2024-09-16T16:39:42.116188+0000","last_unstale":"2024-09-16T16:40:47.305274+0000","last_undegraded":"2024-09-16T16:40:47.305274+0000","last_fullsized":"2024-09-16T16:40:47.305274+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.307777+0000","last_change":"2024-09-16T16:39:44.062407+0000","last_active":"2024-09-16T16:40:47.307777+0000","last_peered":"2024-09-16T16:40:47.307777+0000","last_clean":"2024-09-16T16:40:47.307777+0000","last_became_active":"2024-09-16T16:39:42.148215+0000","last_became_peered":"2024-09-16T16:39:42.148215+0000","last_unstale":"2024-09-16T16:40:47.307777+0000","last_undegraded":"2024-09-16T16:40:47.307777+0000","last_fullsized":"2024-09-16T16:40:47.307777+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.304831+0000","last_change":"2024-09-16T16:39:44.058579+0000","last_active":"2024-09-16T16:40:47.304831+0000","last_peered":"2024-09-16T16:40:47.304831+0000","last_clean":"2024-09-16T16:40:47.304831+0000","last_became_active":"2024-09-16T16:39:42.133685+0000","last_became_peered":"2024-09-16T16:39:42.133685+0000","last_unstale":"2024-09-16T16:40:47.304831+0000","last_undegraded":"2024-09-16T16:40:47.304831+0000","last_fullsized":"2024-09-16T16:40:47.304831+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":28,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.309114+0000","last_change":"2024-09-16T16:39:44.064255+0000","last_active":"2024-09-16T16:40:47.309114+0000","last_peered":"2024-09-16T16:40:47.309114+0000","last_clean":"2024-09-16T16:40:47.309114+0000","last_became_active":"2024-09-16T16:39:42.094003+0000","last_became_peered":"2024-09-16T16:39:42.094003+0000","last_unstale":"2024-09-16T16:40:47.309114+0000","last_undegraded":"2024-09-16T16:40:47.309114+0000","last_fullsized":"2024-09-16T16:40:47.309114+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":96,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.305730+0000","last_change":"2024-09-16T16:39:41.078026+0000","last_active":"2024-09-16T16:40:47.305730+0000","last_peered":"2024-09-16T16:40:47.305730+0000","last_clean":"2024-09-16T16:40:47.305730+0000","last_became_active":"2024-09-16T16:39:41.066265+0000","last_became_peered":"2024-09-16T16:39:41.066265+0000","last_unstale":"2024-09-16T16:40:47.305730+0000","last_undegraded":"2024-09-16T16:40:47.305730+0000","last_fullsized":"2024-09-16T16:40:47.305730+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:40.005960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:40.005960+0000","last_clean_scrub_stamp":"2024-09-16T16:39:40.005960+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4131,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":359,"num_read_kb":689,"num_write":175,"num_write_kb":677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":8262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":180,"ondisk_log_size":180,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29990,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26980,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344860,"statfs":{"total":96636764160,"available":96609136640,"internally_reserved":0,"allocated":151552,"data_stored":40131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27052,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344788,"statfs":{"total":96636764160,"available":96609062912,"internally_reserved":0,"allocated":225280,"data_stored":634630,"data_compressed":29990,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27052,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344788,"statfs":{"total":96636764160,"available":96609062912,"internally_reserved":0,"allocated":225280,"data_stored":634630,"data_compressed":29990,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29990,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":4131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":4131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T16:41:18.774 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T16:41:18.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:41:18.884+0000 7f28e64bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:41:18.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:41:18.892+0000 7f28e64bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:41:19.085 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:41:19.086 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-16T16:41:19.099 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2024-09-16T16:41:19.017396+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594499,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":412,"num_read_kb":735,"num_write":254,"num_write_kb":1783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":232,"ondisk_log_size":232,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":81084,"kb_used_data":588,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034436,"statfs":{"total":289910292480,"available":289827262464,"internally_reserved":0,"allocated":602112,"data_stored":1309391,"data_compressed":59980,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4112,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":359,"num_read_kb":689,"num_write":173,"num_write_kb":675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003022"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.304933+0000","last_change":"2024-09-16T16:39:44.053256+0000","last_active":"2024-09-16T16:40:47.304933+0000","last_peered":"2024-09-16T16:40:47.304933+0000","last_clean":"2024-09-16T16:40:47.304933+0000","last_became_active":"2024-09-16T16:39:42.085202+0000","last_became_peered":"2024-09-16T16:39:42.085202+0000","last_unstale":"2024-09-16T16:40:47.304933+0000","last_undegraded":"2024-09-16T16:40:47.304933+0000","last_fullsized":"2024-09-16T16:40:47.304933+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.311794+0000","last_change":"2024-09-16T16:39:44.063464+0000","last_active":"2024-09-16T16:40:47.311794+0000","last_peered":"2024-09-16T16:40:47.311794+0000","last_clean":"2024-09-16T16:40:47.311794+0000","last_became_active":"2024-09-16T16:39:42.115450+0000","last_became_peered":"2024-09-16T16:39:42.115450+0000","last_unstale":"2024-09-16T16:40:47.311794+0000","last_undegraded":"2024-09-16T16:40:47.311794+0000","last_fullsized":"2024-09-16T16:40:47.311794+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"14'169","reported_seq":375,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.307691+0000","last_change":"2024-09-16T16:39:44.064037+0000","last_active":"2024-09-16T16:40:47.307691+0000","last_peered":"2024-09-16T16:40:47.307691+0000","last_clean":"2024-09-16T16:40:47.307691+0000","last_became_active":"2024-09-16T16:39:42.129388+0000","last_became_peered":"2024-09-16T16:39:42.129388+0000","last_unstale":"2024-09-16T16:40:47.307691+0000","last_undegraded":"2024-09-16T16:40:47.307691+0000","last_fullsized":"2024-09-16T16:40:47.307691+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4112,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":187,"num_read_kb":674,"num_write":169,"num_write_kb":673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"18'8","reported_seq":93,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.335859+0000","last_change":"2024-09-16T16:39:44.064988+0000","last_active":"2024-09-16T16:40:47.335859+0000","last_peered":"2024-09-16T16:40:47.335859+0000","last_clean":"2024-09-16T16:40:47.335859+0000","last_became_active":"2024-09-16T16:39:42.135711+0000","last_became_peered":"2024-09-16T16:39:42.135711+0000","last_unstale":"2024-09-16T16:40:47.335859+0000","last_undegraded":"2024-09-16T16:40:47.335859+0000","last_fullsized":"2024-09-16T16:40:47.335859+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":163,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":38,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.305274+0000","last_change":"2024-09-16T16:39:44.053534+0000","last_active":"2024-09-16T16:40:47.305274+0000","last_peered":"2024-09-16T16:40:47.305274+0000","last_clean":"2024-09-16T16:40:47.305274+0000","last_became_active":"2024-09-16T16:39:42.116188+0000","last_became_peered":"2024-09-16T16:39:42.116188+0000","last_unstale":"2024-09-16T16:40:47.305274+0000","last_undegraded":"2024-09-16T16:40:47.305274+0000","last_fullsized":"2024-09-16T16:40:47.305274+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.307777+0000","last_change":"2024-09-16T16:39:44.062407+0000","last_active":"2024-09-16T16:40:47.307777+0000","last_peered":"2024-09-16T16:40:47.307777+0000","last_clean":"2024-09-16T16:40:47.307777+0000","last_became_active":"2024-09-16T16:39:42.148215+0000","last_became_peered":"2024-09-16T16:39:42.148215+0000","last_unstale":"2024-09-16T16:40:47.307777+0000","last_undegraded":"2024-09-16T16:40:47.307777+0000","last_fullsized":"2024-09-16T16:40:47.307777+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.304831+0000","last_change":"2024-09-16T16:39:44.058579+0000","last_active":"2024-09-16T16:40:47.304831+0000","last_peered":"2024-09-16T16:40:47.304831+0000","last_clean":"2024-09-16T16:40:47.304831+0000","last_became_active":"2024-09-16T16:39:42.133685+0000","last_became_peered":"2024-09-16T16:39:42.133685+0000","last_unstale":"2024-09-16T16:40:47.304831+0000","last_undegraded":"2024-09-16T16:40:47.304831+0000","last_fullsized":"2024-09-16T16:40:47.304831+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":28,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.309114+0000","last_change":"2024-09-16T16:39:44.064255+0000","last_active":"2024-09-16T16:40:47.309114+0000","last_peered":"2024-09-16T16:40:47.309114+0000","last_clean":"2024-09-16T16:40:47.309114+0000","last_became_active":"2024-09-16T16:39:42.094003+0000","last_became_peered":"2024-09-16T16:39:42.094003+0000","last_unstale":"2024-09-16T16:40:47.309114+0000","last_undegraded":"2024-09-16T16:40:47.309114+0000","last_fullsized":"2024-09-16T16:40:47.309114+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":96,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.305730+0000","last_change":"2024-09-16T16:39:41.078026+0000","last_active":"2024-09-16T16:40:47.305730+0000","last_peered":"2024-09-16T16:40:47.305730+0000","last_clean":"2024-09-16T16:40:47.305730+0000","last_became_active":"2024-09-16T16:39:41.066265+0000","last_became_peered":"2024-09-16T16:39:41.066265+0000","last_unstale":"2024-09-16T16:40:47.305730+0000","last_undegraded":"2024-09-16T16:40:47.305730+0000","last_fullsized":"2024-09-16T16:40:47.305730+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:40.005960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:40.005960+0000","last_clean_scrub_stamp":"2024-09-16T16:39:40.005960+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4131,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":359,"num_read_kb":689,"num_write":175,"num_write_kb":677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":8262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":180,"ondisk_log_size":180,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29990,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26980,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344860,"statfs":{"total":96636764160,"available":96609136640,"internally_reserved":0,"allocated":151552,"data_stored":40131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27052,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344788,"statfs":{"total":96636764160,"available":96609062912,"internally_reserved":0,"allocated":225280,"data_stored":634630,"data_compressed":29990,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27052,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344788,"statfs":{"total":96636764160,"available":96609062912,"internally_reserved":0,"allocated":225280,"data_stored":634630,"data_compressed":29990,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29990,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":4131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":4131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T16:41:19.099 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T16:41:19.100 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T16:41:19.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:41:19.209+0000 7f013b991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:41:19.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:41:19.217+0000 7f013b991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:41:19.412 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:41:19.412 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-16T16:41:19.425 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2024-09-16T16:41:19.017396+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594499,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":412,"num_read_kb":735,"num_write":254,"num_write_kb":1783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":232,"ondisk_log_size":232,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":81084,"kb_used_data":588,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034436,"statfs":{"total":289910292480,"available":289827262464,"internally_reserved":0,"allocated":602112,"data_stored":1309391,"data_compressed":59980,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4112,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":359,"num_read_kb":689,"num_write":173,"num_write_kb":675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003022"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.304933+0000","last_change":"2024-09-16T16:39:44.053256+0000","last_active":"2024-09-16T16:40:47.304933+0000","last_peered":"2024-09-16T16:40:47.304933+0000","last_clean":"2024-09-16T16:40:47.304933+0000","last_became_active":"2024-09-16T16:39:42.085202+0000","last_became_peered":"2024-09-16T16:39:42.085202+0000","last_unstale":"2024-09-16T16:40:47.304933+0000","last_undegraded":"2024-09-16T16:40:47.304933+0000","last_fullsized":"2024-09-16T16:40:47.304933+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.311794+0000","last_change":"2024-09-16T16:39:44.063464+0000","last_active":"2024-09-16T16:40:47.311794+0000","last_peered":"2024-09-16T16:40:47.311794+0000","last_clean":"2024-09-16T16:40:47.311794+0000","last_became_active":"2024-09-16T16:39:42.115450+0000","last_became_peered":"2024-09-16T16:39:42.115450+0000","last_unstale":"2024-09-16T16:40:47.311794+0000","last_undegraded":"2024-09-16T16:40:47.311794+0000","last_fullsized":"2024-09-16T16:40:47.311794+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"14'169","reported_seq":375,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.307691+0000","last_change":"2024-09-16T16:39:44.064037+0000","last_active":"2024-09-16T16:40:47.307691+0000","last_peered":"2024-09-16T16:40:47.307691+0000","last_clean":"2024-09-16T16:40:47.307691+0000","last_became_active":"2024-09-16T16:39:42.129388+0000","last_became_peered":"2024-09-16T16:39:42.129388+0000","last_unstale":"2024-09-16T16:40:47.307691+0000","last_undegraded":"2024-09-16T16:40:47.307691+0000","last_fullsized":"2024-09-16T16:40:47.307691+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4112,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":187,"num_read_kb":674,"num_write":169,"num_write_kb":673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"18'8","reported_seq":93,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.335859+0000","last_change":"2024-09-16T16:39:44.064988+0000","last_active":"2024-09-16T16:40:47.335859+0000","last_peered":"2024-09-16T16:40:47.335859+0000","last_clean":"2024-09-16T16:40:47.335859+0000","last_became_active":"2024-09-16T16:39:42.135711+0000","last_became_peered":"2024-09-16T16:39:42.135711+0000","last_unstale":"2024-09-16T16:40:47.335859+0000","last_undegraded":"2024-09-16T16:40:47.335859+0000","last_fullsized":"2024-09-16T16:40:47.335859+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":163,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":38,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.305274+0000","last_change":"2024-09-16T16:39:44.053534+0000","last_active":"2024-09-16T16:40:47.305274+0000","last_peered":"2024-09-16T16:40:47.305274+0000","last_clean":"2024-09-16T16:40:47.305274+0000","last_became_active":"2024-09-16T16:39:42.116188+0000","last_became_peered":"2024-09-16T16:39:42.116188+0000","last_unstale":"2024-09-16T16:40:47.305274+0000","last_undegraded":"2024-09-16T16:40:47.305274+0000","last_fullsized":"2024-09-16T16:40:47.305274+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.307777+0000","last_change":"2024-09-16T16:39:44.062407+0000","last_active":"2024-09-16T16:40:47.307777+0000","last_peered":"2024-09-16T16:40:47.307777+0000","last_clean":"2024-09-16T16:40:47.307777+0000","last_became_active":"2024-09-16T16:39:42.148215+0000","last_became_peered":"2024-09-16T16:39:42.148215+0000","last_unstale":"2024-09-16T16:40:47.307777+0000","last_undegraded":"2024-09-16T16:40:47.307777+0000","last_fullsized":"2024-09-16T16:40:47.307777+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.304831+0000","last_change":"2024-09-16T16:39:44.058579+0000","last_active":"2024-09-16T16:40:47.304831+0000","last_peered":"2024-09-16T16:40:47.304831+0000","last_clean":"2024-09-16T16:40:47.304831+0000","last_became_active":"2024-09-16T16:39:42.133685+0000","last_became_peered":"2024-09-16T16:39:42.133685+0000","last_unstale":"2024-09-16T16:40:47.304831+0000","last_undegraded":"2024-09-16T16:40:47.304831+0000","last_fullsized":"2024-09-16T16:40:47.304831+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":28,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.309114+0000","last_change":"2024-09-16T16:39:44.064255+0000","last_active":"2024-09-16T16:40:47.309114+0000","last_peered":"2024-09-16T16:40:47.309114+0000","last_clean":"2024-09-16T16:40:47.309114+0000","last_became_active":"2024-09-16T16:39:42.094003+0000","last_became_peered":"2024-09-16T16:39:42.094003+0000","last_unstale":"2024-09-16T16:40:47.309114+0000","last_undegraded":"2024-09-16T16:40:47.309114+0000","last_fullsized":"2024-09-16T16:40:47.309114+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":96,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.305730+0000","last_change":"2024-09-16T16:39:41.078026+0000","last_active":"2024-09-16T16:40:47.305730+0000","last_peered":"2024-09-16T16:40:47.305730+0000","last_clean":"2024-09-16T16:40:47.305730+0000","last_became_active":"2024-09-16T16:39:41.066265+0000","last_became_peered":"2024-09-16T16:39:41.066265+0000","last_unstale":"2024-09-16T16:40:47.305730+0000","last_undegraded":"2024-09-16T16:40:47.305730+0000","last_fullsized":"2024-09-16T16:40:47.305730+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:40.005960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:40.005960+0000","last_clean_scrub_stamp":"2024-09-16T16:39:40.005960+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4131,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":359,"num_read_kb":689,"num_write":175,"num_write_kb":677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":8262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":180,"ondisk_log_size":180,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29990,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26980,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344860,"statfs":{"total":96636764160,"available":96609136640,"internally_reserved":0,"allocated":151552,"data_stored":40131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27052,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344788,"statfs":{"total":96636764160,"available":96609062912,"internally_reserved":0,"allocated":225280,"data_stored":634630,"data_compressed":29990,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27052,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344788,"statfs":{"total":96636764160,"available":96609062912,"internally_reserved":0,"allocated":225280,"data_stored":634630,"data_compressed":29990,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29990,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":4131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":4131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T16:41:19.426 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T16:41:19.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:41:19.535+0000 7f727b2fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:41:19.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:41:19.543+0000 7f727b2fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:41:19.740 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:41:19.740 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":18,"fsid":"3da74dbb-4758-44e9-b920-58aeda43326b","created":"2024-09-16T16:39:30.256938+0000","modified":"2024-09-16T16:40:47.286987+0000","last_up_change":"2024-09-16T16:39:39.005066+0000","last_in_change":"2024-09-16T16:39:32.180029+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T16:39:39.015271+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T16:39:40.101450+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c51754b3-b226-47a9-942c-c1de35bbfcee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6805","nonce":637735179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":637735179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6803","nonce":637735179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":637735179}]},"public_addr":"172.21.15.26:6805/637735179","cluster_addr":"172.21.15.26:6804/637735179","heartbeat_back_addr":"172.21.15.26:6803/637735179","heartbeat_front_addr":"172.21.15.26:6802/637735179","state":["exists","up"]},{"osd":1,"uuid":"dd91df32-3f85-43ce-b983-0b5c1b40059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":3366424101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6807","nonce":3366424101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6809","nonce":3366424101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3366424101}]},"public_addr":"172.21.15.26:6806/3366424101","cluster_addr":"172.21.15.26:6807/3366424101","heartbeat_back_addr":"172.21.15.26:6809/3366424101","heartbeat_front_addr":"172.21.15.26:6810/3366424101","state":["exists","up"]},{"osd":2,"uuid":"01c4163c-223f-415b-ba3b-2ab6c501e706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":241914171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6811","nonce":241914171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6813","nonce":241914171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":241914171}]},"public_addr":"172.21.15.26:6808/241914171","cluster_addr":"172.21.15.26:6811/241914171","heartbeat_back_addr":"172.21.15.26:6813/241914171","heartbeat_front_addr":"172.21.15.26:6814/241914171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:0/364689746":"2024-09-16T17:40:46.586089+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T16:41:20.754 INFO:tasks.ceph:Scrubbing osd.0 2024-09-16T16:41:20.754 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T16:41:20.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:41:20.879+0000 7f80fd111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:41:20.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:41:20.892+0000 7f80fd111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:41:20.909 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T16:41:20.909 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "" 2024-09-16T16:41:20.909 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T16:41:20.921 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-16T16:41:21.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:41:21.030+0000 7f0632897640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:41:21.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:41:21.039+0000 7f0632897640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:41:21.240 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T16:41:21.259 INFO:tasks.ceph:Scrubbing osd.1 2024-09-16T16:41:21.259 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T16:41:21.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:41:21.368+0000 7f86dc663640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:41:21.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:41:21.381+0000 7f86dc663640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:41:21.396 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T16:41:21.397 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "" 2024-09-16T16:41:21.397 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T16:41:21.407 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-16T16:41:21.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:41:21.517+0000 7f6512351640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:41:21.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:41:21.526+0000 7f6512351640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:41:21.726 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T16:41:21.745 INFO:tasks.ceph:Scrubbing osd.2 2024-09-16T16:41:21.745 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T16:41:21.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:41:21.856+0000 7f19d4751640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:41:21.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:41:21.869+0000 7f19d4751640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:41:21.885 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T16:41:21.885 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "" 2024-09-16T16:41:21.885 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T16:41:21.897 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-16T16:41:22.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:41:22.006+0000 7feb43591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:41:22.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:41:22.015+0000 7feb43591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:41:22.219 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T16:41:22.238 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T16:41:22.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:41:22.347+0000 7f8107b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:41:22.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:41:22.356+0000 7f8107b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:41:22.548 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:41:22.549 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-16T16:41:22.561 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-09-16T16:41:21.017775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594499,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":412,"num_read_kb":735,"num_write":254,"num_write_kb":1783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":232,"ondisk_log_size":232,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":81084,"kb_used_data":588,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034436,"statfs":{"total":289910292480,"available":289827262464,"internally_reserved":0,"allocated":602112,"data_stored":1309391,"data_compressed":59980,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002918"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.304933+0000","last_change":"2024-09-16T16:39:44.053256+0000","last_active":"2024-09-16T16:40:47.304933+0000","last_peered":"2024-09-16T16:40:47.304933+0000","last_clean":"2024-09-16T16:40:47.304933+0000","last_became_active":"2024-09-16T16:39:42.085202+0000","last_became_peered":"2024-09-16T16:39:42.085202+0000","last_unstale":"2024-09-16T16:40:47.304933+0000","last_undegraded":"2024-09-16T16:40:47.304933+0000","last_fullsized":"2024-09-16T16:40:47.304933+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.311794+0000","last_change":"2024-09-16T16:39:44.063464+0000","last_active":"2024-09-16T16:40:47.311794+0000","last_peered":"2024-09-16T16:40:47.311794+0000","last_clean":"2024-09-16T16:40:47.311794+0000","last_became_active":"2024-09-16T16:39:42.115450+0000","last_became_peered":"2024-09-16T16:39:42.115450+0000","last_unstale":"2024-09-16T16:40:47.311794+0000","last_undegraded":"2024-09-16T16:40:47.311794+0000","last_fullsized":"2024-09-16T16:40:47.311794+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"14'169","reported_seq":375,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.307691+0000","last_change":"2024-09-16T16:39:44.064037+0000","last_active":"2024-09-16T16:40:47.307691+0000","last_peered":"2024-09-16T16:40:47.307691+0000","last_clean":"2024-09-16T16:40:47.307691+0000","last_became_active":"2024-09-16T16:39:42.129388+0000","last_became_peered":"2024-09-16T16:39:42.129388+0000","last_unstale":"2024-09-16T16:40:47.307691+0000","last_undegraded":"2024-09-16T16:40:47.307691+0000","last_fullsized":"2024-09-16T16:40:47.307691+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4112,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":187,"num_read_kb":674,"num_write":169,"num_write_kb":673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"18'8","reported_seq":93,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.335859+0000","last_change":"2024-09-16T16:39:44.064988+0000","last_active":"2024-09-16T16:40:47.335859+0000","last_peered":"2024-09-16T16:40:47.335859+0000","last_clean":"2024-09-16T16:40:47.335859+0000","last_became_active":"2024-09-16T16:39:42.135711+0000","last_became_peered":"2024-09-16T16:39:42.135711+0000","last_unstale":"2024-09-16T16:40:47.335859+0000","last_undegraded":"2024-09-16T16:40:47.335859+0000","last_fullsized":"2024-09-16T16:40:47.335859+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":163,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":38,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.305274+0000","last_change":"2024-09-16T16:39:44.053534+0000","last_active":"2024-09-16T16:40:47.305274+0000","last_peered":"2024-09-16T16:40:47.305274+0000","last_clean":"2024-09-16T16:40:47.305274+0000","last_became_active":"2024-09-16T16:39:42.116188+0000","last_became_peered":"2024-09-16T16:39:42.116188+0000","last_unstale":"2024-09-16T16:40:47.305274+0000","last_undegraded":"2024-09-16T16:40:47.305274+0000","last_fullsized":"2024-09-16T16:40:47.305274+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.307777+0000","last_change":"2024-09-16T16:39:44.062407+0000","last_active":"2024-09-16T16:40:47.307777+0000","last_peered":"2024-09-16T16:40:47.307777+0000","last_clean":"2024-09-16T16:40:47.307777+0000","last_became_active":"2024-09-16T16:39:42.148215+0000","last_became_peered":"2024-09-16T16:39:42.148215+0000","last_unstale":"2024-09-16T16:40:47.307777+0000","last_undegraded":"2024-09-16T16:40:47.307777+0000","last_fullsized":"2024-09-16T16:40:47.307777+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.304831+0000","last_change":"2024-09-16T16:39:44.058579+0000","last_active":"2024-09-16T16:40:47.304831+0000","last_peered":"2024-09-16T16:40:47.304831+0000","last_clean":"2024-09-16T16:40:47.304831+0000","last_became_active":"2024-09-16T16:39:42.133685+0000","last_became_peered":"2024-09-16T16:39:42.133685+0000","last_unstale":"2024-09-16T16:40:47.304831+0000","last_undegraded":"2024-09-16T16:40:47.304831+0000","last_fullsized":"2024-09-16T16:40:47.304831+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":28,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.309114+0000","last_change":"2024-09-16T16:39:44.064255+0000","last_active":"2024-09-16T16:40:47.309114+0000","last_peered":"2024-09-16T16:40:47.309114+0000","last_clean":"2024-09-16T16:40:47.309114+0000","last_became_active":"2024-09-16T16:39:42.094003+0000","last_became_peered":"2024-09-16T16:39:42.094003+0000","last_unstale":"2024-09-16T16:40:47.309114+0000","last_undegraded":"2024-09-16T16:40:47.309114+0000","last_fullsized":"2024-09-16T16:40:47.309114+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:41.006717+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":96,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:40:47.305730+0000","last_change":"2024-09-16T16:39:41.078026+0000","last_active":"2024-09-16T16:40:47.305730+0000","last_peered":"2024-09-16T16:40:47.305730+0000","last_clean":"2024-09-16T16:40:47.305730+0000","last_became_active":"2024-09-16T16:39:41.066265+0000","last_became_peered":"2024-09-16T16:39:41.066265+0000","last_unstale":"2024-09-16T16:40:47.305730+0000","last_undegraded":"2024-09-16T16:40:47.305730+0000","last_fullsized":"2024-09-16T16:40:47.305730+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:39:40.005960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:39:40.005960+0000","last_clean_scrub_stamp":"2024-09-16T16:39:40.005960+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4131,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":359,"num_read_kb":689,"num_write":175,"num_write_kb":677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":8262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":180,"ondisk_log_size":180,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29990,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26980,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344860,"statfs":{"total":96636764160,"available":96609136640,"internally_reserved":0,"allocated":151552,"data_stored":40131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27052,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344788,"statfs":{"total":96636764160,"available":96609062912,"internally_reserved":0,"allocated":225280,"data_stored":634630,"data_compressed":29990,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27052,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344788,"statfs":{"total":96636764160,"available":96609062912,"internally_reserved":0,"allocated":225280,"data_stored":634630,"data_compressed":29990,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29990,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":4131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":4131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T16:41:22.565 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T16:39:41.006717+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=39, tm_sec=41, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=41, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T16:41:22.565 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T16:39:41.006717+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=39, tm_sec=41, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=41, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T16:41:22.565 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T16:39:41.006717+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=39, tm_sec=41, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=41, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T16:41:22.566 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T16:39:41.006717+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=39, tm_sec=41, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=41, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T16:41:22.566 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T16:39:41.006717+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=39, tm_sec=41, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=41, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T16:41:22.566 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T16:39:41.006717+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=39, tm_sec=41, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=41, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T16:41:22.566 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T16:39:41.006717+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=39, tm_sec=41, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=41, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T16:41:22.566 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T16:39:41.006717+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=39, tm_sec=41, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=41, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T16:41:22.566 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T16:39:40.005960+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=39, tm_sec=40, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=16, tm_min=41, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T16:41:22.566 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T16:41:42.567 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T16:41:42.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:41:42.692+0000 7fc4fb846640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:41:42.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:41:42.701+0000 7fc4fb846640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:41:42.901 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T16:41:42.901 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-16T16:41:42.914 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2024-09-16T16:41:41.021713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":501,"num_read_kb":741,"num_write":260,"num_write_kb":1786,"num_scrub_errors":0,"num_shallow_scrub_errors":2,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":251,"ondisk_log_size":251,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":81068,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034452,"statfs":{"total":289910292480,"available":289827278848,"internally_reserved":0,"allocated":585728,"data_stored":1301167,"data_compressed":59980,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4112,"num_objects":-3,"num_object_clones":0,"num_object_copies":-6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":89,"num_read_kb":6,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":2,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002463"},"pg_stats":[{"pgid":"2.7","version":"18'1","reported_seq":29,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:41:21.731614+0000","last_change":"2024-09-16T16:41:21.744508+0000","last_active":"2024-09-16T16:41:21.731614+0000","last_peered":"2024-09-16T16:41:21.731614+0000","last_clean":"2024-09-16T16:41:21.731614+0000","last_became_active":"2024-09-16T16:39:42.085202+0000","last_became_peered":"2024-09-16T16:39:42.085202+0000","last_unstale":"2024-09-16T16:41:21.731614+0000","last_undegraded":"2024-09-16T16:41:21.731614+0000","last_fullsized":"2024-09-16T16:41:21.731614+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'1","last_scrub_stamp":"2024-09-16T16:41:21.744312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:41:21.744312+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"18'2","reported_seq":29,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:41:21.736691+0000","last_change":"2024-09-16T16:41:21.755035+0000","last_active":"2024-09-16T16:41:21.736691+0000","last_peered":"2024-09-16T16:41:21.736691+0000","last_clean":"2024-09-16T16:41:21.736691+0000","last_became_active":"2024-09-16T16:39:42.115450+0000","last_became_peered":"2024-09-16T16:39:42.115450+0000","last_unstale":"2024-09-16T16:41:21.736691+0000","last_undegraded":"2024-09-16T16:41:21.736691+0000","last_fullsized":"2024-09-16T16:41:21.736691+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2024-09-16T16:41:21.754829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:41:21.754829+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"18'171","reported_seq":379,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:41:21.735396+0000","last_change":"2024-09-16T16:41:21.757114+0000","last_active":"2024-09-16T16:41:21.735396+0000","last_peered":"2024-09-16T16:41:21.735396+0000","last_clean":"2024-09-16T16:41:21.735396+0000","last_became_active":"2024-09-16T16:39:42.129388+0000","last_became_peered":"2024-09-16T16:39:42.129388+0000","last_unstale":"2024-09-16T16:41:21.735396+0000","last_undegraded":"2024-09-16T16:41:21.735396+0000","last_fullsized":"2024-09-16T16:41:21.735396+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'171","last_scrub_stamp":"2024-09-16T16:41:21.756919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:41:21.756919+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":187,"num_read_kb":674,"num_write":171,"num_write_kb":673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"18'17","reported_seq":134,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:41:21.734357+0000","last_change":"2024-09-16T16:41:21.760120+0000","last_active":"2024-09-16T16:41:21.734357+0000","last_peered":"2024-09-16T16:41:21.734357+0000","last_clean":"2024-09-16T16:41:21.734357+0000","last_became_active":"2024-09-16T16:39:42.135711+0000","last_became_peered":"2024-09-16T16:39:42.135711+0000","last_unstale":"2024-09-16T16:41:21.734357+0000","last_undegraded":"2024-09-16T16:41:21.734357+0000","last_fullsized":"2024-09-16T16:41:21.734357+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'17","last_scrub_stamp":"2024-09-16T16:41:21.759918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:41:21.759918+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":242,"num_read_kb":20,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":1,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":44,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:41:21.243011+0000","last_change":"2024-09-16T16:41:21.256282+0000","last_active":"2024-09-16T16:41:21.243011+0000","last_peered":"2024-09-16T16:41:21.243011+0000","last_clean":"2024-09-16T16:41:21.243011+0000","last_became_active":"2024-09-16T16:39:42.116188+0000","last_became_peered":"2024-09-16T16:39:42.116188+0000","last_unstale":"2024-09-16T16:41:21.243011+0000","last_undegraded":"2024-09-16T16:41:21.243011+0000","last_fullsized":"2024-09-16T16:41:21.243011+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"14'2","last_scrub_stamp":"2024-09-16T16:41:21.256079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:41:21.256079+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":29,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:41:22.224043+0000","last_change":"2024-09-16T16:41:22.235099+0000","last_active":"2024-09-16T16:41:22.224043+0000","last_peered":"2024-09-16T16:41:22.224043+0000","last_clean":"2024-09-16T16:41:22.224043+0000","last_became_active":"2024-09-16T16:39:42.148215+0000","last_became_peered":"2024-09-16T16:39:42.148215+0000","last_unstale":"2024-09-16T16:41:22.224043+0000","last_undegraded":"2024-09-16T16:41:22.224043+0000","last_fullsized":"2024-09-16T16:41:22.224043+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T16:41:22.234918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:41:22.234918+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"18'1","reported_seq":29,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:41:22.222515+0000","last_change":"2024-09-16T16:41:22.232544+0000","last_active":"2024-09-16T16:41:22.222515+0000","last_peered":"2024-09-16T16:41:22.222515+0000","last_clean":"2024-09-16T16:41:22.222515+0000","last_became_active":"2024-09-16T16:39:42.133685+0000","last_became_peered":"2024-09-16T16:39:42.133685+0000","last_unstale":"2024-09-16T16:41:22.222515+0000","last_undegraded":"2024-09-16T16:41:22.222515+0000","last_fullsized":"2024-09-16T16:41:22.222515+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'1","last_scrub_stamp":"2024-09-16T16:41:22.232236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:41:22.232236+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"18'5","reported_seq":38,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:41:21.731137+0000","last_change":"2024-09-16T16:41:21.748890+0000","last_active":"2024-09-16T16:41:21.731137+0000","last_peered":"2024-09-16T16:41:21.731137+0000","last_clean":"2024-09-16T16:41:21.731137+0000","last_became_active":"2024-09-16T16:39:42.094003+0000","last_became_peered":"2024-09-16T16:39:42.094003+0000","last_unstale":"2024-09-16T16:41:21.731137+0000","last_undegraded":"2024-09-16T16:41:21.731137+0000","last_fullsized":"2024-09-16T16:41:21.731137+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'5","last_scrub_stamp":"2024-09-16T16:41:21.748693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:41:21.748693+0000","last_clean_scrub_stamp":"2024-09-16T16:39:41.006717+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":1,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":98,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T16:41:21.729511+0000","last_change":"2024-09-16T16:41:21.761029+0000","last_active":"2024-09-16T16:41:21.729511+0000","last_peered":"2024-09-16T16:41:21.729511+0000","last_clean":"2024-09-16T16:41:21.729511+0000","last_became_active":"2024-09-16T16:39:41.066265+0000","last_became_peered":"2024-09-16T16:39:41.066265+0000","last_unstale":"2024-09-16T16:41:21.729511+0000","last_undegraded":"2024-09-16T16:41:21.729511+0000","last_fullsized":"2024-09-16T16:41:21.729511+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"11'52","last_scrub_stamp":"2024-09-16T16:41:21.760807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T16:41:21.760807+0000","last_clean_scrub_stamp":"2024-09-16T16:39:40.005960+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":448,"num_read_kb":695,"num_write":181,"num_write_kb":680,"num_scrub_errors":0,"num_shallow_scrub_errors":2,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29990,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705783,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26980,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344860,"statfs":{"total":96636764160,"available":96609136640,"internally_reserved":0,"allocated":151552,"data_stored":40131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738487,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27044,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344796,"statfs":{"total":96636764160,"available":96609071104,"internally_reserved":0,"allocated":217088,"data_stored":630518,"data_compressed":29990,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738487,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27044,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344796,"statfs":{"total":96636764160,"available":96609071104,"internally_reserved":0,"allocated":217088,"data_stored":630518,"data_compressed":29990,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29990,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T16:41:42.915 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-16T16:41:43.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:41:43.020+0000 7f8ddf6d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:41:43.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T16:41:43.032+0000 7f8ddf6d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T16:41:43.255 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T16:41:43.255 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T16:41:43.256 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T16:41:43.256 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T16:41:43.311 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T16:41:43.311 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T16:41:43.311 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T16:41:43.368 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T16:41:43.368 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T16:41:43.369 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T16:41:43.427 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T16:41:43.428 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T16:41:43.428 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-16T16:41:43.428 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T16:41:43.455 INFO:tasks.ceph.mgr.x:Stopped 2024-09-16T16:41:43.455 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T16:41:43.456 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T16:41:43.456 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T16:41:43.508 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T16:41:43.508 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T16:41:43.509 DEBUG:teuthology.orchestra.run.smithi026:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-16T16:41:43.577 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi026.front.sepia.ceph.com 2024-09-16T16:41:43.577 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-16T16:41:43.766 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi026.front.sepia.ceph.com 2024-09-16T16:41:43.766 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-16T16:41:43.840 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi026.front.sepia.ceph.com 2024-09-16T16:41:43.840 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-16T16:41:43.899 INFO:tasks.ceph:Archiving mon data... 2024-09-16T16:41:43.900 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2024-09-16_16:23:48-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi/7907402/data/mon.a.tgz 2024-09-16T16:41:43.900 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2024-09-16T16:41:43.915 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.KZvTdaGP8n 2024-09-16T16:41:43.915 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.KZvTdaGP8n 2024-09-16T16:41:44.020 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.KZvTdaGP8n 2024-09-16T16:41:44.104 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.KZvTdaGP8n is 164KB 2024-09-16T16:41:44.157 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.KZvTdaGP8n 2024-09-16T16:41:44.172 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T16:41:44.172 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T16:41:44.293 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T16:41:44.293 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T16:41:44.294 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-09-16_16:23:48-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi/7907402/remote/smithi026/crash 2024-09-16T16:41:44.295 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T16:41:44.358 INFO:tasks.ceph:Compressing logs... 2024-09-16T16:41:44.358 DEBUG:teuthology.orchestra.run.smithi026:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T16:41:44.423 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-16T16:41:44.423 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29271.log 2024-09-16T16:41:44.443 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29698.log 2024-09-16T16:41:44.443 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.29271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30698.log 2024-09-16T16:41:44.443 INFO:teuthology.orchestra.run.smithi026.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29271.log.gz 2024-09-16T16:41:44.444 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31380.log 2024-09-16T16:41:44.444 INFO:teuthology.orchestra.run.smithi026.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29698.log.gz 2024-09-16T16:41:44.444 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29933.log 2024-09-16T16:41:44.444 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30698.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30698.log.gz 2024-09-16T16:41:44.444 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31088.log 2024-09-16T16:41:44.444 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31380.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31380.log.gz 2024-09-16T16:41:44.444 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31046.log 2024-09-16T16:41:44.444 INFO:teuthology.orchestra.run.smithi026.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29933.log.gz 2024-09-16T16:41:44.444 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31088.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-16T16:41:44.444 INFO:teuthology.orchestra.run.smithi026.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31088.log.gz 2024-09-16T16:41:44.444 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31214.log 2024-09-16T16:41:44.444 INFO:teuthology.orchestra.run.smithi026.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31046.log.gz 2024-09-16T16:41:44.445 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29849.log 2024-09-16T16:41:44.445 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31256.log 2024-09-16T16:41:44.445 INFO:teuthology.orchestra.run.smithi026.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31214.log.gz 2024-09-16T16:41:44.445 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30573.log 2024-09-16T16:41:44.445 INFO:teuthology.orchestra.run.smithi026.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29849.log.gz 2024-09-16T16:41:44.445 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28912.log 2024-09-16T16:41:44.445 INFO:teuthology.orchestra.run.smithi026.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31256.log.gz 2024-09-16T16:41:44.445 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30590.log 2024-09-16T16:41:44.445 INFO:teuthology.orchestra.run.smithi026.stderr: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.30573.log.gz 2024-09-16T16:41:44.445 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.28912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30183.log 2024-09-16T16:41:44.445 INFO:teuthology.orchestra.run.smithi026.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28912.log.gz 2024-09-16T16:41:44.445 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30590.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T16:41:44.446 INFO:teuthology.orchestra.run.smithi026.stderr: 60.3% -- replaced with /var/log/ceph/ceph-client.admin.30590.log.gz 2024-09-16T16:41:44.446 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30057.log 2024-09-16T16:41:44.446 INFO:teuthology.orchestra.run.smithi026.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30183.log.gz 2024-09-16T16:41:44.446 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28824.log 2024-09-16T16:41:44.446 INFO:teuthology.orchestra.run.smithi026.stderr: 86.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T16:41:44.446 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29112.log 2024-09-16T16:41:44.446 INFO:teuthology.orchestra.run.smithi026.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30057.log.gz 2024-09-16T16:41:44.446 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30878.log 2024-09-16T16:41:44.446 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.28824.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28824.log.gz 2024-09-16T16:41:44.446 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31298.log 2024-09-16T16:41:44.446 INFO:teuthology.orchestra.run.smithi026.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29112.log.gz 2024-09-16T16:41:44.446 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30878.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30878.log.gz 2024-09-16T16:41:44.446 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31004.log 2024-09-16T16:41:44.447 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29754.log 2024-09-16T16:41:44.447 INFO:teuthology.orchestra.run.smithi026.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31298.log.gz 2024-09-16T16:41:44.447 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29301.log 2024-09-16T16:41:44.447 INFO:teuthology.orchestra.run.smithi026.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31004.log.gz 2024-09-16T16:41:44.447 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29754.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29754.log.gz 2024-09-16T16:41:44.447 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30768.log 2024-09-16T16:41:44.447 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29301.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-16T16:41:44.447 INFO:teuthology.orchestra.run.smithi026.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29301.log.gz 2024-09-16T16:41:44.447 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29891.log 2024-09-16T16:41:44.447 INFO:teuthology.orchestra.run.smithi026.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.30768.log.gz 2024-09-16T16:41:44.447 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30920.log 2024-09-16T16:41:44.447 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30143.log 2024-09-16T16:41:44.447 INFO:teuthology.orchestra.run.smithi026.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29891.log.gz 2024-09-16T16:41:44.447 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29022.log 2024-09-16T16:41:44.447 INFO:teuthology.orchestra.run.smithi026.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30920.log.gz 2024-09-16T16:41:44.448 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-16T16:41:44.448 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30143.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30143.log.gz 2024-09-16T16:41:44.448 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30182.log 2024-09-16T16:41:44.448 INFO:teuthology.orchestra.run.smithi026.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29022.log.gz 2024-09-16T16:41:44.448 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30631.log 2024-09-16T16:41:44.448 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30056.log 2024-09-16T16:41:44.448 INFO:teuthology.orchestra.run.smithi026.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30182.log.gz 2024-09-16T16:41:44.448 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30631.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T16:41:44.448 INFO:teuthology.orchestra.run.smithi026.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.30631.log.gz 2024-09-16T16:41:44.448 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30056.log: gzip -5 --verbose -- 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30056.log.gz 2024-09-16T16:41:44.448 INFO:teuthology.orchestra.run.smithi026.stderr: /var/log/ceph/ceph-client.admin.30746.log 2024-09-16T16:41:44.448 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-16T16:41:44.448 INFO:teuthology.orchestra.run.smithi026.stderr: 86.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T16:41:44.448 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30746.log: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.30746.log.gz 2024-09-16T16:41:44.449 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27459.log 2024-09-16T16:41:44.449 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30653.log 2024-09-16T16:41:44.449 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph.tmp-client.admin.27459.log: gzip -5 --verbose -- 68.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27459.log.gz 2024-09-16T16:41:44.449 INFO:teuthology.orchestra.run.smithi026.stderr: /var/log/ceph/ceph-client.admin.30591.log 2024-09-16T16:41:44.449 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30653.log: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.30653.log.gz 2024-09-16T16:41:44.449 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29062.log 2024-09-16T16:41:44.449 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29779.log 2024-09-16T16:41:44.449 INFO:teuthology.orchestra.run.smithi026.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.30591.log.gz 2024-09-16T16:41:44.449 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31172.log 2024-09-16T16:41:44.449 INFO:teuthology.orchestra.run.smithi026.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29062.log.gz 2024-09-16T16:41:44.449 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30962.log 2024-09-16T16:41:44.449 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29779.log: /var/log/ceph/ceph-client.admin.31172.log: gzip -5 --verbose 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31172.log.gz 2024-09-16T16:41:44.449 INFO:teuthology.orchestra.run.smithi026.stderr: -- /var/log/ceph/ceph-client.admin.31130.log 2024-09-16T16:41:44.449 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30962.log: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.29779.log.gz 2024-09-16T16:41:44.450 INFO:teuthology.orchestra.run.smithi026.stderr: 50.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30353.log 2024-09-16T16:41:44.450 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30962.log.gz 2024-09-16T16:41:44.450 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31130.log: gzip -5 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31130.log.gz 2024-09-16T16:41:44.450 INFO:teuthology.orchestra.run.smithi026.stderr: --verbose -- /var/log/ceph/ceph-client.admin.30676.log 2024-09-16T16:41:44.450 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30353.log: 49.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30722.log 2024-09-16T16:41:44.450 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30353.log.gz 2024-09-16T16:41:44.450 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30055.log 2024-09-16T16:41:44.450 INFO:teuthology.orchestra.run.smithi026.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.30676.log.gz 2024-09-16T16:41:44.450 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29975.log 2024-09-16T16:41:44.450 INFO:teuthology.orchestra.run.smithi026.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30722.log.gz 2024-09-16T16:41:44.450 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30311.log 2024-09-16T16:41:44.450 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30055.log: /var/log/ceph/ceph-client.admin.29975.log: gzip -5 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29975.log.gz 2024-09-16T16:41:44.451 INFO:teuthology.orchestra.run.smithi026.stderr: --verbose -- /var/log/ceph/ceph-client.admin.31340.log 2024-09-16T16:41:44.451 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30311.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30055.log.gz 2024-09-16T16:41:44.451 INFO:teuthology.orchestra.run.smithi026.stderr: 50.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30269.log 2024-09-16T16:41:44.451 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30311.log.gz 2024-09-16T16:41:44.451 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31340.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31340.log.gz 2024-09-16T16:41:44.451 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30269.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30269.log.gz 2024-09-16T16:41:44.456 INFO:teuthology.orchestra.run.smithi026.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-16T16:41:44.464 INFO:teuthology.orchestra.run.smithi026.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-16T16:41:44.480 INFO:teuthology.orchestra.run.smithi026.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-16T16:41:44.502 INFO:teuthology.orchestra.run.smithi026.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-16T16:41:44.603 INFO:teuthology.orchestra.run.smithi026.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-16T16:41:44.605 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T16:41:44.605 INFO:teuthology.orchestra.run.smithi026.stderr:real 0m0.192s 2024-09-16T16:41:44.605 INFO:teuthology.orchestra.run.smithi026.stderr:user 0m0.369s 2024-09-16T16:41:44.605 INFO:teuthology.orchestra.run.smithi026.stderr:sys 0m0.053s 2024-09-16T16:41:44.606 INFO:tasks.ceph:Archiving logs... 2024-09-16T16:41:44.606 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/log/ceph to /home/teuthworker/archive/matan-2024-09-16_16:23:48-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi/7907402/remote/smithi026/log 2024-09-16T16:41:44.607 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T16:41:44.805 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T16:41:44.819 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T16:41:44.819 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T16:41:44.874 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T16:41:44.882 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T16:41:44.882 DEBUG:teuthology.orchestra.run.smithi026:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T16:41:44.902 INFO:teuthology.orchestra.run.smithi026.stderr:bash: line 1: ntpq: command not found 2024-09-16T16:41:44.906 INFO:teuthology.orchestra.run.smithi026.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T16:41:44.906 INFO:teuthology.orchestra.run.smithi026.stdout:=============================================================================== 2024-09-16T16:41:44.906 INFO:teuthology.orchestra.run.smithi026.stdout:^* hv01.front.sepia.ceph.com 3 6 377 59 -617us[ -609us] +/- 66ms 2024-09-16T16:41:44.906 INFO:teuthology.orchestra.run.smithi026.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 57 -64us[ -64us] +/- 52ms 2024-09-16T16:41:44.906 INFO:teuthology.orchestra.run.smithi026.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 58 +764us[ +764us] +/- 72ms 2024-09-16T16:41:44.907 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T16:41:44.907 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T16:41:44.917 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T16:41:44.918 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T16:41:44.944 DEBUG:teuthology.orchestra.run.smithi026:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T16:41:45.011 DEBUG:teuthology.orchestra.run.smithi026:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T16:41:45.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T16:41:45.039 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T16:41:45.047 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T16:41:45.085 INFO:teuthology.task.internal:Duration was 564.746521 seconds 2024-09-16T16:41:45.086 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T16:41:45.122 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T16:41:45.122 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T16:41:45.162 INFO:teuthology.orchestra.run.smithi026.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T16:41:45.483 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T16:41:45.484 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi026.front.sepia.ceph.com 2024-09-16T16:41:45.484 DEBUG:teuthology.orchestra.run.smithi026:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T16:41:45.546 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T16:41:45.546 DEBUG:teuthology.orchestra.run.smithi026:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T16:41:45.614 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T16:41:45.614 DEBUG:teuthology.orchestra.run.smithi026:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T16:41:45.757 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T16:41:45.767 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T16:41:45.767 DEBUG:teuthology.orchestra.run.smithi026:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T16:41:45.793 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T16:41:45.801 DEBUG:teuthology.orchestra.run.smithi026:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:41:45.857 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = core 2024-09-16T16:41:45.871 DEBUG:teuthology.orchestra.run.smithi026:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T16:41:45.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T16:41:45.931 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T16:41:45.941 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T16:41:45.942 DEBUG:teuthology.misc:Transferring archived files from smithi026:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2024-09-16_16:23:48-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi/7907402/remote/smithi026 2024-09-16T16:41:45.942 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T16:41:46.005 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T16:41:46.005 DEBUG:teuthology.orchestra.run.smithi026:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T16:41:46.055 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T16:41:46.065 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T16:41:46.065 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T16:41:46.095 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T16:41:46.095 DEBUG:teuthology.orchestra.run.smithi026:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T16:41:46.111 INFO:teuthology.orchestra.run.smithi026.stdout: 265340 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 16:41 /home/ubuntu/cephtest 2024-09-16T16:41:46.112 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T16:41:46.129 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T16:41:46.197 INFO:teuthology.run:Summary data: description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore tasks/rbd_lock_and_fence} duration: 564.7465207576752 flavor: crimson owner: scheduled_matan@teuthology success: true 2024-09-16T16:41:46.197 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T16:41:46.279 INFO:teuthology.run:pass